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Research and Evaluation at The Annie E. Casey Foundation 

The Annie E. Casey Foundation’s mission, motivation, and message to the world is its commitment to 
changing and improving life outcomes for our most disadvantaged children and families. That belief is 
rooted in the conviction that outcomes for children will not improve without fundamental, 
comprehensive, and durable changes in many service and support systems. Current health, education, 
juvenile justice, and other delivery systems for disadvantaged children and families too often are 
fragmented, inaccessible, expensive, and irrelevant. They frequently fail to deliver essential services 
until it is too late, contributing to an overall level of ineffectiveness and to an intergenerational cycle of 
poverty. 

The Foundation operates on the premise that these conditions can be reversed — that communities can 
prosper, families can thrive, and children can develop when neighborhoods are supportive, sustaining, 
and served by systems that are relevant, respectful, and rooted in the communities that they serve. The 
Foundation believes that strategic investments in awareness building, capacity development, program 
demonstrations, and research and evaluation can help move dysfunctional service systems toward 
greater collaboration, coordination, and flexibility. 

In addition to leadership, funding, and other key factors, these changes require accurate, relevant, and 
compelling information. Research and evaluation are a conduit for information — and information is 
power. With information on results provided by evaluation, community stakeholders can make better 
decisions about organizational practices. Similarly, states informed by research and evaluation can 
make better decisions about the allocation of resources and policies that affect children and families. 

Evaluation plays a major role in the Foundation’s theory of change, as a tool for: 

• Improving accountability: contributing to understanding about the degree to which 
interventions represent good judgments about the organizations, communities, and people in 
which the Foundation places its confidence and resources. 

• Revealing the soundness of theories, the practicality of policies, the appropriateness of planning 
timelines, the relevance of technical assistance, and the extent to which the Foundation has 
established effective partnerships with grantees. 

• Informing funders about the viability of working with states, cities, community-based 
organizations, and child- and family-serving systems to achieve real transformation and reform. 

For these reasons, the Casey Foundation believes that research and evaluation can, should, and must be 
a critical and integral component of comprehensive reform strategies. 
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Conference Overview 



T he Annie E. Casey Foundation 

continued its dialogue about difficult 
research issues with leaders and 
evaluators of social interventions at 
a conference in March 1999. The 
conference, held in Baltimore, was the 
fourth in a series of forums offered 
periodically by the Foundation since 1994 
to help identify emerging challenges and 
explore promising solutions. 

The topics of the conferences reflect the 
Foundation’s evolving strategies for 
improving outcomes for disadvantaged 
children and families, as well as new 
developments in the art and science of 
understanding social change. The first 
conference focused on issues related to the 
evaluation of complex public system 
reform. The second conference addressed 
ways of using evaluation results more 
effectively to advance public policy and 
build public will. The third conference 
explored the complexities of evaluating 
comprehensive community change. 

The fourth conference occurred as the 
Foundation launched a new neighborhood 
transformation and family development 
strategy, based on the following premises: 

® Although good interventions and 

service systems are important, even the 
best system-reform initiatives are not 
powerful enough to leverage all of the 
change needed to produce better 
outcomes for children. Children are 
most likely to do well when they have 
strong families — families that are 
resourceful, informed, and able to draw 
on an array of supportive people, 
services, and opportunities. 

• It is possible to improve outcomes for 
children by strengthening their families. 
Families can be strengthened by 



transforming neighborhoods into places 
that nurture and support families and 
reinforce children’s chances for success. 
Neighborhood transformation — 
community building — involves providing 
the incentives, investments, and 
opportunities needed to connect families 
with jobs, friends and neighbors, faith- 
based institutions, and other social 
relationships, networks, and supports. 

It is clear to us that when families ... live in 
places that exacerbate their struggles — 
places that are perceived to be or are 
unsafe, places that are prone to extensive 
drug use and drug commerce, where 
economic investments are weak, where 
social supports are fragile . . . and where 
public systems are disproportionately 
weak — it is easy for even the strongest 
families to feel compromised and 
overwhelmed. 

— Tony Cipollone 
The Annie E. Casey Foundation 



The Foundation’s investments under this 
strategy, including the Making Connections 
demonstration initiative, will seek to build 
public will, advance public policy, reform 
public systems, build family-supporting 
neighborhoods, and strengthen the capacity 
of practitioners, researchers, and 
community stakeholders to engage in this 
work and to use data effectively for 
strategic analysis and planning. 

The 1999 conference was designed to 
stimulate discussion about several issues 
that the foundation will need to address in 
implementing its new strategy. As 
described by conference host and Director 
of Evaluation Tony Cipollone, these issues 
include: 
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• Understanding families and 
neighborhoods, especially (1) how to 
define, foster, and measure family 
strength; (2) how neighborhood 
resources and characteristics can help 
or hinder families’ efforts to nurture 
their children; (3) which child and 
family outcomes are most strongly 
linked to neighborhood factors; (4) 
what types of interventions hold the 
most promise for promoting strong 
families and children in poor 
neighborhoods; and (5) how to make 
family strengthening the core engine 
and outcome of a comprehensive 
neighborhood transformation effort 

• Helping communities analyze and use 
data effectively, especially when 
evaluations seek to blend local and 
national assessments of multi-site 
initiatives; challenges include 
identifying outcomes and merging local 
and national theories of change 

• Accommodating long timeframes for 
interventions in ways that support 
evaluation of long-term endeavors, 
opportunities to reflect on the 
appropriateness of strategies and 
indicators of success, and adherence to 
core goals and principles over time 

• Maximizing the utility of evaluations, 
especially by assessing intangible but 
important elements such as 
neighborhood efficacy or empowerment 
and its impact on child outcomes; 
challenges include (1) assessing the 
roles of local and national funders and 
technical assistance providers, (2) 
establishing a role for evaluators that 
makes them useful to sites without 
compromising their objectivity, and (3) 
creating products that communicate 
findings in useful and compelling ways 



Cross-Cutting Themes 

2. The intersection between 

neighborhood and family is vital to 
children ’s well-being, but research on 
the topic has many gaps. 

Neighborhood and family are not just 
separate variables in child development; 
they interact in important ways. Therefore, 
the relationships between neighborhoods 
and families — the processes and 
mechanisms by which one influences the 
other — ought to command attention from 
researchers and resources from initiatives. 
However, researchers do not yet adequately 
understand how the cultural, racial/ethnic, 
and ecological characteristics of 
neighborhoods affect the way in which 
children experience families and 
communities at various stages of their 
development. Nor do social scientists and 
funders fully understand what mechanisms 
link families and neighborhoods and which 
interventions are most promising for 
developing families and transforming 
neighborhoods. 

2. Although research confirms the sense 
that neighborhood matters to children 
and families, it offers little guidance 
on key concerns for policy makers. 

Current research generally does not explain 
how to help families choose healthy 
neighborhoods, how to improve existing 
neighborhoods to better support families 
and children, or how to help residents of 
distressed neighborhoods avoid or 
overcome the problems that surround them. 
The research base also is sparse regarding 
neighborhood effects on the development of 
young children. 

3. The context and characteristics of 
neighborhoods influence the outcomes 
experienced by children and families. 
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Initiatives designed to improve outcomes 
for children need to understand: why 
neighborhoods and families matter; how 
families mediate neighborhood influences, 
and vice versa; and how to introduce, 
support, and modify crucial neighborhood 
mechanisms to achieve positive outcomes. 

Similarly, researchers must identify the 
neighborhood’s cultural, racial, ecological, 
and socio-economic variables; incorporate 
them in analyses; and, in some cases, make 
them a distinct focus of study. Most 
current research does not explain which 
mechanisms are linked to specific 
outcomes, although some studies have 
described factors that mediate the way in 
which residents perceive or experience 
certain aspects of their neighborhoods. 

4. Efforts to strengthen families and 
neighborhoods must do more than 
simply provide social services or 
external supports — they must build 
local capacity for change and the 
social capital that enables residents to 
sustain and expand improvements. 

Interventions tend to focus on “fixing” 
problems rather than on helping people 
become able to meet their own needs. Real 
family and community enhancement 
requires the development of social 
capital — the resources and opportunities 
that develop capacity and self-sufficiency in 
individuals and local institutions. 

Many researchers and evaluators are 
working to better understand how social 
capital and a community’s collective 
efficacy can strengthen neighborhoods and 
families and how interventions can 
stimulate these key qualities of social 
change. Some interventions — and also 
some evaluations — use technical assistance 
to build local capacity. This can be an 
effective strategy for strengthening 
individuals and institutions if the technical 



assistance targets the specific needs and 
interests of local stakeholders. 

5. Initiatives that strengthen families and 
build communities present several 
challenges for evaluation design and 
methodology. 

These challenges include: (1) defining the 
unit of analysis; (2) clarifying the purpose 
of the evaluation; (3) defining outcomes, 
indicators, and measures for such concepts 
as social capital, family strength, 
community, community efficacy and spirit, 
and empowerment; (4) measuring 
processes, interactions, perceptions, and 
relationships as well as behaviors, 
activities, and outcomes; (5) timing the 
start of evaluation activities to ensure 
smooth data collection and cooperation 
from local partners; (6) integrating 
qualitative and quantitative data; (7) 
presenting data to diverse audiences in 
useful and compelling ways; and (8), in the 
case of evaluations that have local and 
national components, effectively managing 
both tiers of activity. 

This conference summary presents the 
discussion of these themes and other issues. 
Section I, Understanding the Connection 
between Families and Communities, tackles 
the questions, “Does neighborhood 
matter?” and, “How does neighborhood 
matter?” Section II, Evaluating Initiatives 
that Strengthen Families and Build 
Communities , describes the challenges 
facing evaluators of neighborhood- and 
family-building initiatives, issues involved 
in providing technical assistance on 
evaluation, and tactics for presenting data 
in useful and compelling ways. Section III, 
Lessons for Future Initiatives and 
Evaluations, concludes the summary with 
lessons and recommendations proposed by 
the participants. Appendices contain the 
conference agenda, transcripts of two 
featured speeches, and a list of participants. 
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Understanding the Connection Between 
Families and Communities 



W e know a lot about children and how they develop into young people and adults. We 
know a thing or two about what makes a family healthy and strong, and perhaps a bit 
more about the factors that can undermine families. And we know quite a bit about 
communities and what it takes to make them viable, productive, and nurturing. But 
what do we really know about the intersection between children, families, and neighborhoods? 



Does Neighborhood Matter? 

Most Americans assume that growing up in 
a “bad” neighborhood will somehow 
jeopardize a child’s future. Accordingly, 
those who can afford to do so choose to 
live in neighborhoods where their children 
can attend good schools and be surrounded 
by neighbors and peers who share their 
family’s goals and values. Although some 
social scientists contradict this assumption, 
suggesting that bad outcomes prevail in 
certain neighborhoods simply because high- 
risk individuals cluster there, panelist 
Margery Turner described a consensus 
among many researchers that neighborhood 
conditions such as poverty and employment 
rates do significantly affect a range of 
outcomes for children and adults, including 
education, employment, sexual activity, and 
criminal involvement. 

To date, most research on neighborhood 
effects has focused on adolescents and has 
not examined the ways in which 
neighborhood conditions may affect 
children or influence the way in which 
children develop into adolescents. Nor has 
the empirical research provided much 
evidence of what it is that matters most in 
neighborhoods or what the mechanisms are 
by which neighborhood characteristics are 
translated into outcomes for children and 
families. Researchers also have not paid 
much attention to historical and cultural 



factors and to the combined effect of 
multiple factors. 

Despite these gaps, social researchers are 
actively constructing theories and concepts 
of how neighborhoods matter. 



How Does Neighborhood 
Matter? 

Conference participants offered several 
research-based perspectives on how 
neighborhoods matter, including concepts 
like social capital, social networks and 
“embeddedness,” and collective efficacy. 
Participants also identified mechanisms by 
which neighborhoods can affect child and 
family well-being and highlighted the 
importance of neighborhood context in 
child and family development. 

Social Capital 

Although the concept of social capital is not 
new, it has taken on new meanings and 
importance in research on families and 
communities. As described by panelist Bill 
Rohe, social capital refers to the social 
trust, connections, and norms that enable 
individuals to collectively pursue shared 
objectives and solve problems. According 
to this model, which Rohe used in his 
research on social capital in Pittsburgh 
neighborhoods, increasing the amount and 
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quality of neighborhood interaction, or 
civic engagement, should (1) increase 
residents’ trust of each other and of local 
and citywide institutions and (2) build more 
extensive social networks among 
neighborhood residents. These changes 
help residents develop shared norms and 
can lead to more effective problem solving, 
better community institutions, and benefits 
for individuals. Rohe’s research measured 
such factors as volunteer activity in the 
neighborhood, residents’ participation in 
organizations outside the neighborhood, the 
way in which residents used social 
networks, and the extent to which 
community organizations worked together. 

“What’s really new is the combination of 
concepts that are embedded within [social 
capital] and the implied relationships within 
those concepts,” Rohe said. “The concept 
of social capital links civic engagement to 
trust and ultimately to effective institutions, 
and in this sense social capital is really a ... 
causal model. It’s not a unitary construct 
like self-esteem or sense of control.” 

Other models of social capital recognize 
additional factors that influence individual 
and neighborhood outcomes, including 
intergenerational culture (the degree to 
which adults and children in a 
neighborhood relate to each other), 
reciprocity (the knowledge, resources, and 
social supports that people exchange), and 
continuity or stability. 

Research has linked social capital with 
several positive outcomes. The work of 
Robert Putnam, cited by several panelists, 
showed that social capital is associated with 
levels of economic development and the 
effectiveness of political institutions. 

Rohe’s research in Pittsburgh showed that 
neighborhoods with higher levels of social 
capital were more stable over time, as 
indicated by property values. 



Pittsburgh Study Defines and Measures 
Social Capital 

Conference panelist Bill Rohe, Director of 
the Center for Urban and Regional Studies 
at the University of North Carol ina-Chapel 
Hill, and colleague Ken Temkin, 
examined levels of social capital in 
Pittsburgh neighborhoods and its relation- 
ship to neighborhood change over time. 

Using measures of social capital developed 
from a survey conducted by Roger 
Ahlbrandt in 1980, Rohe and Temkin 
surveyed and interviewed residents about 
four topics: community engagement, the 
characteristics of the community’s social 
networks, trust levels within the 
community (of fellow residents and of 
institutions), and the extent and 
effectiveness of the community 
organizations. The researchers then 
merged their survey data with 1980 and 
1990 Census data to see what 
neighborhoods with high levels of social 
capital in 1980 looked like a decade later. 

Rohe and Temkin found that 
neighborhoods with high levels of social 
capital were more likely to remain stable 
over time. 



Social Networks and Embeddedness 

Social networks — the informal ties that 
connect individuals to information and 
resources — exist to varying degrees in most 
neighborhoods. Research by panelist James 
Johnson shows that these “bridging” 
networks give residents a significant boost 
toward positive outcomes by acting as 
conduits of information and assistance from 
one group of individuals to another. 

Johnson’s research found that the extent to 
which a resident is embedded in one or 
several social networks that enhance a 
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specific outcome, such as employment, 
helps to determine how successful the 
individual is in achieving his or her goals 
(in this case, obtaining a job). Johnson 
found that women of all race/ethnicities in 
the Los Angeles neighborhoods he studied 
were more likely to find jobs if they had 
links to people who either lived in other 
neighborhoods, were employed, or were 
educated beyond the high school level. 
Johnson found that the effect of some social 
bridges — gender, race/ethnicity, and 



Social Networks Influence Employment 
Outcomes 

In a study of urban inequality conducted by 
James H. Johnson of the University of 
North Carolina-Chapel Hill, researchers 
used social networks to examine changing 
labor market dynamics and the role of racial 
and ethnic polarization. Researchers 
interviewed and surveyed people from 
8,600 households and 4,000 employers in 
Los Angeles, Boston, Detroit, and Atlanta. 
Respondents were asked to identify three 
people outside their household with whom 
they discussed important matters. For each 
of these people, the survey recorded data on 
race/ethnicity, sex, level of education, 
relationship to the respondent, marital 
status, residence inside or outside of the 
respondent’s neighborhood, employment 
status, and receipt of public assistance. 

From these data, the researchers determined 
that a survey respondent had a “bridging” 
social tie if at least one of the people 
identified by the respondent was of a 
different race/ethnicity, sex, or 
neighborhood, had more than a high school 
education, had a job, or received public 
assistance. The researchers examined these 
“social embeddedness” variables 
individually and in clusters, along with 
family context, to determine their effect on 
the survey respondents’ own employment 
outcomes. 



receipt of public assistance — differed 
according to the respondent’s race/ 
ethnicity. White and Hispanic women were 
more likely to be employed if they had 
networks that included people of both 
sexes, although that finding was not true 
for African-American women. Hispanic 
women who had racially diverse networks 
were more likely to be employed than their 
counterparts with racially homogenous 
networks, although this was not the case for 
women of other races. African-American 
and Hispanic women who had at least one 
person in their network who received 
public assistance were less likely to be 
working than their counterparts who did not 
have this bridge, although this association 
did not occur for white women. 

Johnson cited the following findings by 
other researchers that help to clarify the 
effect of social networks: 

• Network composition, especially 
heterogeneity, affects both attitudes and 
employment outcomes. In a study of 
job outcomes for African Americans, 
Braddock and McPartland found that 
African Americans who were 
embedded in racially segregated 
networks, which contain less beneficial 
information about jobs, had lower 
incomes than their counterparts in 
heterogeneous networks. 

e The strength of network ties (i.e., the 
amount of time invested, emotional 
intensity, intimacy, and reciprocal 
services that characterize ties) 
influences employment outcomes. 
Research by Granovetter, who studied 
white males, found that those who used 
strong network ties to conduct job 
searches earned higher wages. 

• Using a personal contact to obtain a job 
is not necessarily beneficial for 
disadvantaged individuals. In a study 
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of African Americans in Atlanta, 
Browne and Hewitt found that those 
people who acquired their jobs through 
personal contacts were more likely to 
have racially segregated, and therefore 
lower-paying, jobs than those who used 
formal means. 

o Social networks may mediate the 
potentially negative effects of a 
neighborhood’s inadequate social 
capital. Ethnographic research by 
Robin Jarrett has shown that the 
children of some poor women whose 
social networks connect them with 
institutions beyond their own 
neighborhoods are more likely to have 
high aspirations for education and other 
“community-bridging behaviors” than 
their peers who are not embedded in 
bridging social networks. Conversely, 
the absence of social networks can 
hinder a person’s efforts to obtain 
employment and meet other goals. 

The social networks that produce the best 
outcomes have many members, include 
people of diverse social statuses who 
operate in several fields of activity, and 
bridge distinct social worlds, according to 
Johnson. Some types of social networks 
are more valuable than others for achieving 
certain goals. For individuals who want to 
find employment, being embedded in a 
network that bridges neighborhoods or 
provides links to employed people is most 
useful; links to people of the opposite sex 
are more important than links to people of 
another race. 



Collective Efficacy 

The concept of collective efficacy described 
by keynote speaker Felton Earls focuses on 
the extent to which an organization — in this 
case, a neighborhood — is ready to solve a 
problem or address a goal. The model of 



collective efficacy used in Earls’ research 
incorporates measures of intergenerational 
culture, reciprocity, and continuity, and 
differs from the concept of social capital by 
placing a special emphasis on a 
community’s potential to mobilize for 
action to resolve problems. 



Chicago Study Builds Understanding 
of Collective Efficacy 

The Project on Human Development, led 
by Dr. Felton Earls of the Harvard 
School of Public Health, integrates 
research on community context with 
research on human psychological and 
physical development. The study 
focuses on Chicago because it 
encompasses established, new, and 
transitional neighborhoods that house an 
array of new immigrants as well as 
European- and African-American 
residents. 

The project, which began in 1995, 
blends a study of the characteristics and 
contexts of 343 communities with a 
longitudinal study of 6,000 children 
between the ages of birth and 18 and a 
separate sample of about 9,000 
individuals. Researchers collect data 
through surveys of residents and key 
players who interact with neighborhoods, 
to learn about perceptions and behaviors, 
and by videotaping the physical 
environment and social interactions. 



In the collective efficacy model, residents’ 
perceptions of and attitudes toward each 
other are as important as characteristics 
such as race or socioeconomic class, if not 
more so. In the Chicago neighborhoods 
where he assessed collective efficacy, Earls 
asked residents such questions as, “If you 
saw a child skipping school, do you think 
people around here would do something 
about it?” and, “If a fire station in the 
neighborhood was about to close, do you 
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think people around here would do 
something about it?” 

Earls found that communities with high 
levels of social capital have lower levels of 
crime, violence, teen pregnancy, and 
delinquency. Further, he found that a 
neighborhood can have high levels of 
collective efficacy even if most residents do 
not actually know each other. In other 
words, residents may believe that their 
neighbors will act a certain way, even 
though they are strangers. 

Mechanisms by which Neighborhoods 
Affect Child and Family Development 
and Well-Being 

Panelist Margery Turner, director of the 
Metropolitan Housing and Community 
Policy Center at the Urban Institute, 
provided an overview and critique of the 
research literature on neighborhood effects 
on children and families. With colleague 
Ingrid Gould-Ellen, Turner reviewed social 
science research to learn “what has been 
empirically established, what those findings 
mean for policy and for interventions in 
neighborhoods, and what additional 
research might be needed in this area.” 

She found evidence of some independent 
neighborhood effects at virtually every 
stage of a person’s life and over a wide 
range of domains, including education, 
employment, sexual activity, and criminal 
involvement. 

Turner summarized the key mechanisms 
identified by research as follows: 

• The quality of local public services, 
such as elementary schools, police 
departments, recreational facilities, 
child care, and health care 

• Socialization by adults outside the 
family with whom children interact; 
these individuals include role models, 



communal disciplinarians, and adults 
who help children understand what 
constitutes normal and acceptable 
behavior 

• Peer influences, especially aspirations 
for education and employment 

• Social networks, which help define 
norms of behavior and connect 
individuals with jobs and opportunities 

• Exposure to crime and violence, 
which traumatizes the emotional and 
intellectual development of young 
children and creates opportunities for 
older children to get hurt or into trouble 

• Physical distance and isolation, 
especially from jobs and transportation 

® The stressors that residents encounter 
(both real and perceived), which can 
either motivate or undermine an 
individual; the balance between 
stressors and supports available to 
children and families, and the 
individuals’ ability to recognize, 
accept, obtain, and use supports, also 
affect outcomes 

• Contextual factors that trigger coping 
mechanisms, especially short-term 
self-protective behaviors that may 
undermine long-term positive 
development 

These mechanisms are not mutually 
exclusive; all, in combination with other 
factors, probably play a role in determining 
how neighborhoods matter. 

The Role of Community Context 

The context in which a child and family 
lives — especially the availability of 
resources and the presence of real or 
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perceived danger — and the way in which 
people view that context can either enhance 
or debilitate child and family development. 

In particular, perceptions about community 
context influence how a person behaves. In 
a study of 600 adolescent African-American 
males living in urban neighborhoods, 
panelist Margaret Beale Spencer found that 
they reacted to their high-risk environments 
by exhibiting a heightened bravado. In 
some ways, this was an effective short-term 
coping mechanism: It left the young men 
constantly prepared to respond to perceived 
and actual threats. The youth’s bravado 
also placed them at greater risk with people 
outside their neighborhood, however. 

Neighborhood conditions also influence 
some family management practices. A 
study of parenting practices in low- and 
middle-income Philadelphia families, 

Research on Identity and Resilience 
Recognizes the Importance of Context 

Margaret Spencer conducts research on 
Philadelphia’s Start on Success Program, 
which provides emotionally disturbed 
youth with classes at the University of 
Pennsylvania, service learning, job skills 
development, and mentoring. The 
program uses family involvement to 
provide a supportive context for students’ 
development and teaches students that they 
have positive options for responding to 
negative environments. Parents participate 
in group exercises, and mentors serve as 
surrogate parents. 

Spencer assesses the students’ recognition 
of supports that offset stress. “This 
youngster, as he or she acquires skills, is 
enveloped by individuals who are vested in 
positive movement through the program,” 
she said. Students develop “ways of 
perceiving environments that leaves them 
open to using more adaptive coping 
mechanisms.” 



conducted by Frank Furstenberg and 
described at the conference by Claudia 
Coulton, found that although the 
psychological aspects of parenting did not 
differ significantly across neighborhoods, 
parents who lived in the most diminished 
neighborhoods — areas characterized by few 
resources and little trust, reciprocity, and 
other elements of social capital — generally 
were more restrictive of their children and 
had to work extra hard to connect the 
children with organizations and 
opportunities outside the neighborhood. 



Research Shows that Neighborhood 
Context Affects Parenting 

In research on 500 families in low- to 
middle-income neighborhoods in 
Philadelphia, Frank Furstenburg addressed 
the question, Do parents respond to 
neighborhood conditions in such a way that 
they change their parenting practices ? The 
communities ranged from stable working 
class environments to rapidly declining 
neighborhoods. Furstenburg and his team 
administered telephone surveys to all 
families and conducted an ethnographic 
study in three neighborhoods — one that 
encompassed a very dangerous public 
housing development, one characterized by 
transition and strife, and one that was very 
stable, with many community institutions. 

Furstenburg hypothesized that in 
neighborhoods with low social capital and 
high crime, parents would be very strict 
with their children, while in neighborhoods 
with high social cohesion, parenting styles 
would be more relaxed. The researchers 
found that parenting did differ by 
neighborhood and that the quality of the 
neighborhood made it more or less feasible 
for parents to connect their children with 
high-quality institutions and programs. 
However, the effect of neighborhood 
differences on children’s mental health and 
behavioral outcomes was less clear. 
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Parents who lived in environments with 
greater social cohesion and resources were 
more likely to allow their children to move 
freely within the community and did not 
have to search as hard for opportunities that 
could help their children experience 
positive outcomes. 

Contextual factors are only partially 
mediated by parents and have an important 
effect on when and how a child’s needs will 
be met, according to panelist Robert 
Halpern. For example, some of the after- 
school programs studied by Halpern 
provide children with safe, protected 
environments, supervision by caring adults, 
and positive support such as homework 
assistance and exposure to the arts, sports, 

Research Looks at Children’s Experiences 
in Non-Family Contexts 

Robert Halpern evaluated the Wallace- 
Reader’s Digest Funds’ MOST initiative 
(Making the Most of Out of School Time), 
an after-school program that operates in 
Boston, Chicago, and Seattle. The 
evaluation examines the characteristics of 
after-school participants, the nature of their 
participation, and the nature and quality of 
the MOST sites. Evaluation methods 
included interviews, observation, analysis of 
existing quantitative data on service supply 
in each city, and review of reports from the 
lead agencies participating in the initiative. 

Halpern found that the programs vary 
widely in terms of structure, content, and 
areas of focus, and that this variation carries 
over into children’s experiences. Many 
programs serving low-income children 
demonstrated poor quality. “The trick in 
after-school programs seems to be the 
deliberately designed developmental settings 
that balance safety, a measure of 
supervision and attention, [and] opportunity 
to try [new things] and enjoy a range of arts 
and sports, in spaces that children ... feel 
that they own,” Halpern concluded. 



service learning, and other stimulating 
experiences. A child whose neighborhood 
context includes a high-quality after-school 
program may develop social skills, make 
academic progress, and form important 
bonds with adults, even if his or her home 
environment does not support these 
outcomes. However, contextual factors 
cannot replace important family functions. 
“It will be important to keep expectations 
of after-school programs commensurate 
with their nature as developmental 
supports,” Halpern warned. 

What does the role of context imply for 
interventions and research? Understanding 
the contextual factors that drive children’s 
behavior helps interventions address the 
specific stressors within neighborhoods 
and, consequently, have a greater chance of 
producing desired outcomes. And paying 
attention to context in data collection and 
analysis helps researchers understand how 
neighborhoods matter to children and 
families. 

Community Capacity for Social 
Change 

Since one of the goals of research and 
evaluation is to provide the information 
needed to help plan, manage, assess, 
improve, and advocate for community 
change, it’s important to understand how 
communities engage in social change and 
what capacities are necessary to support 
community-building efforts. Keynote 
speaker Angela Blackwell, director of an 
advocacy institution called PolicyLink, and 
other participants identified the following 
essential qualities and principles of 
community building: 

Community building is an approach, 

not an agenda. 
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The techniques used to build community 
capacity help prepare residents to achieve a 
variety of goals. These tactics are not an 
agenda in themselves; rather, they can be 
applied to almost any plan for change, 
whether it focuses on reducing teen 
pregnancy, revitalizing community housing, 
improving health care services, moving 
people from public assistance to the 
workforce, or any other specific outcome. 



When I say “community building, ” what 
I’m talking about [is] ... continuous self- 
renewing efforts by residents and 
professionals to engage in collective action, 
problem solving, and enrichment that 
results in improved lives; new and 
strengthened assets, relationships, and 
institutions; and new standards and 
expectations for life in communities. 

— Angela Blackwell 



Community building involves 
continuous problem solving and 
collective action. 

Community building seeks to establish 
relationships and ways of working 
collectively that will outlast short-term 
projects, growing deeper and broader until 
all of the people connected to a community 
are participating in problem-solving and 
enrichment efforts. “We need to think of 
these communities as places that have a 
continuous, renewing capacity to educate, 
to make healthy, to employ, so that those 
things get embedded ... [and are] there for 
the next wave of people who come 
through,” Blackwell said. 

Community building creates new 
standards and expectations. 

These higher standards can relate to 
institutions, such as schools; to groups of 
people who play key roles in 
neighborhoods, such as police; or to 



expectations that residents hold for each 
other’s behavior and for their ability to hold 
each other accountable. Such expectations 
strengthen the community relationships, 
institutions, and assets available to children 
and families. 

Effective leadership is crucial. 

Strong leaders have broad visions for 
continuous improvement, visions that are 
driven by an awareness of neighborhood 
needs and strengths. Good leaders motivate 
people, even those who fail to see the full 
potential for change. “When you’re talking 
about achieving things that haven’t been 
achieved before, you need somebody 
leading that effort who inspires people to 
go on in the face of no evidence that [it] 
will get you anywhere,” Blackwell said. 
Effective leaders also capture the interest of 
people outside their own sphere of work, 
which creates valuable opportunities for 
information sharing and reflection. 

The institutions involved in 
community building must have the 
capacity to collaborate across 
organizations, across disciplines, and 
across differences of opinion. 

No single agency is good enough or deep 
enough to mount a truly comprehensive, 
sustainable community-building endeavor 
on its own, Blackwell said. Agencies must 
collaborate, which requires the institutional 
capacity to: 

o Maintain firm goals for change without 
alienating other agencies 

o Introduce good ideas for change in 
ways that encourage community 
members to embrace them as their own 

° Create an environment in which diverse 
partners work together, regardless of 
their differences 
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Effective community-building efforts 
take a comprehensive approach to 
neighborhood change. 

Community-building strategies need to 
bridge the roles and jurisdictions of 
individual people, institutions, services, and 
resources in order to solve the challenges 
faced by children and families. “You have 
to integrate people-based and place-based 
strategies in order to make a difference,” 
observed Ron Register, director of the 
Cleveland Community Building Initiative. 
“You have to think about education at the 
same time you think about things like 
health, and so on, so that you have a 
comprehensive, holistic, integrated 
approach to neighborhood change.” 

The need for comprehensiveness should 
also drive collaborators beyond racial and 
economic boundaries, Blackwell said. 

“You cannot solve the problems in the 
African-American communities in this 
country only working with people who are 
African-American and who happen to live 
in those neighborhoods,” Blackwell said. 
“You have to reach out ... into different 
races, into different classes.” 

Community-building efforts should be 
oriented toward improving assets, 
rather than “fixing” deficits. 

Even the most economically impoverished 
neighborhoods have assets, especially in 
their residents, that can be organized to 
support social change. Focusing on local 
assets acknowledges the positive ingredients 
for community change that already exist 
and helps to ensure that interventions build 
on the neighborhood’s unique resources and 
conditions. The Cleveland Community 
Building Initiative, for example, made asset 
orientation a guiding principle of poverty 
intervention. Two of the four village 
councils created by the initiative began to 
develop inventories of assets relevant to the 
neighborhoods’ action projects. 



Residents and other informed 
stakeholders must be active and 
meaningful participants in planning 
and implementing community- 
building efforts. 

Essential knowledge about communities 
comes from residents, local organizations 
that work in neighborhoods, and other 
people who link neighborhoods with outside 
resources. Often, community-building 
initiatives include these representatives on 
collaborative boards that oversee strategic 
planning; the Neighborhood and Family 
Initiative, described by evaluator Robert 
Chaskin, is one such example. “In order to 
get a comprehensive neighborhood 
development strategy, you need to ground it 
in the priorities and needs and 
understanding of the community,” Chaskin 
explained. The roles, relationships, and 
processes used by these collaboratives offer 
an insight for evaluators into the dynamics 
of community change. 

Programs that operate within 
community-building initiatives should 
have realistic expectations for what 
outcomes can and should be achieved. 

For example, after-school programs — an 
increasingly common approach to 
augmenting children’s education and 
serving families’ child care needs in 
communities with few other 
resources — often target youth development 
as a primary goal. Many of these programs 
do provide activities that let children 
experience success, form bonds with caring 
adults, and become exposed to worlds and 
futures beyond their neighborhood. But, as 
evaluator Robert Halpern observed, it may 
be asking too much of these programs to 
assume that participation “can or should 
nurture the basic skills, sense of worth, 
confidence, or acceptance that family, 
school, and other key basic institutions are 
responsible for providing.” 
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Evaluation is an important part of 
community-building endeavors. 

Evaluation data help communities and 
program funders understand what they have 
accomplished, what they have learned in 
the process, and how they might refine 
their efforts to produce even better 
outcomes. Evaluation data also become a 
valuable tool for advocacy — or, as 



Blackwell describes it, a community 
commodity. “Being able to articulate the 
role and the value of data as a community 
commodity that is understood, sought, and 
utilized is a very important part of the 
change process,” said Blackwell, whose 
work with the Urban Strategies Council to 
help communities use data is described on 
page 44 of this summary. 
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Evaluating Initiatives that 
Strengthen Families and Build Communities 



C omprehensive community initiatives, especially those designed to strengthen families 
and communities, are complex undertakings. They involve intangible, ever-changing 
relationships, interactions, and processes, which makes them as hard to evaluate as they 
are to implement. This section describes issues in three areas central to research on these 
initiatives: Evaluation design and methods, technical assistance to intervention sites, and the 
presentation of research data. 



Challenges for Evaluation 
Design and Methodology 

Major challenges include: (1) clarifying 
the unit of analysis; (2) clarifying the 
purpose and magnitude of the evaluation; 
(3) defining outcomes, indicators, and 
measures; (4) measuring elusive aspects of 
change; (5) conducting comparative 
research; (6) timing evaluation activities; 

(7) integrating qualitative and quantitative 
data; and (8) using a two-tiered approach to 
combine national and local evaluations. 



Clarifying the Unit of Analysis 

What constitutes a neighborhood? Is it a 
place bounded by geographical limits? A 
community historically identified by certain 
characteristics, including demographic 
similarities and social cohesion? A 
uniquely named area perceived by 
residents, and reflected in local culture, as 
having a distinct identity? Or is it a tract or 
a block group designated by the U.S. 

Census or local government? 

The definition of neighborhood used by an 
evaluator holds implications for research 
design, data collection, and analysis. 
Researchers must ensure that their data 
accurately captures effects within the 
individuals, families, and institutions in the 
appropriate area. In addition, interesting 
effects that appear at very small levels, 



such as the block group, can wash out in a 
larger level of analysis. As Felton Earls 
observed, for example, a large public 
housing complex may represent a unique 
ecological niche — essentially, a 
neighborhood — but it also is embedded in a 
larger neighborhood, and it would be 
misleading to equate growing up in the 
public housing complex with growing up a 
block away. 

The best definition of neighborhood for 
measuring change may fluctuate according 
to the effect being measured. As 
Constancia Warren, an evaluator of the 
New York City Beacons initiative, noted, if 
the goal of defining “community” is to 
establish demographic context, the 
neighborhood boundary will probably be 
determined by zip code or Census tract, 
since that is the format most data bases use 
to record demographic information. If the 
purpose is to measure community 
perceptions of an institution, a reasonable 
boundary might be a 10-minute radius from 
the institution being studied. If the purpose 
is to assess changes in neighborhood safety 
attributable to the institution, the area might 
be much smaller, perhaps one to three 
blocks. 

If the unit of analysis is several 
neighborhoods, as it often is in national 
evaluations of multi-site initiatives, 
researchers struggle with two issues: 
overcoming variation within neighborhoods 
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to understand differences between 
neighborhoods, and drawing conclusions 
that cut across neighborhood-specific 
findings. 

In the first instance, a neighborhood that on 
the surface appears to represent a single 
characteristic may, when examined person 
by person or family by family, reveal so 
much variation that it is difficult to make 
useful comparisons with another 
neighborhood. For example, Felton Earls 
described comparing two neighborhoods 
that represented very different levels of 
real-estate value. When he examined real- 
estate value within each neighborhood, he 
also found extensive variation in housing 
costs, even though other aspects of the 
neighborhood might have remained the 
same. “The within- [neighborhood] 
variation is so much ... that you almost say, 
‘Why doesn’t everybody just move into [the 
low-cost area] and save rent,”’ Earls 
observed. While differences across 
neighborhoods are often obvious, the issue 
of distinguishing within-neighborhood 
differences remains a problem for 
researchers, he added. 

The second issue occurs when individual 
neighborhoods have great leeway in 
designing their local transformation, so that 
the actions taken vary tremendously across 
sites. The challenge is to distill coherent 
findings rather than to simply describe 
many discrete changes. One solution in 
this situation is to make the initiative itself 
the unit of analysis, rather than the 
neighborhood — an approach used by the 
evaluation of the Neighborhood and Family 
Initiative, conducted by The Chapin Hall 
Center for Children at the University of 
Chicago. In this case, researchers focused 
on the development of overall goals, ideas, 
principles, and structures rather than on 
neighborhood-specific changes. This 
approach can generate useful information 
about the process of community 



transformation, but it yields fewer lessons 
about some effects. 

Clarifying the Purpose and Magnitude of 
an Evaluation 

Evaluations are conducted to document 
change, assess strategies, expand 
knowledge within and across disciplines, 
inform practitioners, monitor program 
quality, guide policy makers — and 
sometimes all of the above, simultaneously. 
The purpose of an evaluation influences the 
scope of the overall research and the 
amount and type of research than can 
realistically be expected during the first 
years of an initiative. If the purpose is to 
contribute information on a complex 
initiative to a field of knowledge — a broad 
goal that assumes a large public audience 
and demands a large-scale evaluation 
effort — it may not be realistic to expect too 
many early results. If the purpose is to 
help the initiative understand its actions and 
impacts related to one or two specific 
outcomes, or to document a specific 
process, evaluators can more easily produce 
short-term, targeted tools for 
understanding. Some evaluators keep the 
story of an initiative to themselves during 
the course of their work, divulging it at the 
end of the study in a comprehensive report. 
Others envision evaluation as a chance to 
engage sites in a continuous process of 
feedback, reflection, and renewal. 

Several issues complicate efforts to clarify 
an evaluation’s purpose. First, the purpose 
of the initiative often is unclear during 
planning and early implementation. It takes 
some time for local leaders and their 
funders to reach consensus on agendas, 
work out differences, develop relationships, 
move from vague objectives to concrete 
tactics, and organize actions. Until the 
initiative has clear goals, evaluators cannot 
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begin measuring progress toward meeting 
the goals. 

Second, funders of initiatives often are 
deliberately non-prescriptive in their early 
interactions with grantees, hoping to 
stimulate the sites to define their own goals. 
At the same time, grant recipients are 
anxious to know what their funders expect 
to happen; they want to understand the 
rules of the game. As technical assistance 
provider Bill Traynor suggested, “There’s 
as certain deliberateness to the [confusion] 
at the beginning that I think is somewhat 
inevitable and unavoidable, and of course 
that has tremendous impacts ... [on] the 
way that an initiative can be 
operationalized.” That confusion also 
influences how evaluators assess those 
operations and their effects. 



Defining Outcomes. Indicators, and 
Measures 

Evaluations rely on indicators, and 
measures of those indicators, to show 
whether interventions have produced their 
intended outcomes. Evaluations of 
initiatives that involve such concepts as 
social capital, family strength, community, 
community efficacy, and empowerment 
face extra challenges in defining outcomes, 
indicators, and measures. 



Definition of Terms 

Outcomes are the effects caused by an 
intervention’s activities and strategies. 

Indicators are specific, measurable 
manifestations of an outcome. 

Measures are the tools used to collect 
data on indicators and outcomes. 



One problem is that there are no standard 
definitions for many of these concepts, and 
therefore no consistent norms, measures, 
thresholds, or ranges to apply. This makes 
it hard to accurately measure neighborhood 
capacity, compare across sites or 
interventions, and understand successful 
outcomes. 

In addition, changes in neighborhoods and 
individuals take many forms, and it may 
require a combination of evaluation 
approaches to capture the various 
outcomes. For example, evaluators could 
focus on changes in neighborhood 
infrastructure, such as the number, type, 
size, diversity, effectiveness, and 
participation rates of organizations. Or, 
they could measure changes in processes, 
such as collaboration among those 
organizations. Researchers could examine 
engagement and interaction within a 
neighborhood, such as the amount and 
quality of interaction among residents, 
volunteer activity, participation in 
neighborhood organizations, and use of 
neighborhood facilities. Or they could 
focus on engagement across 
neighborhoods, including participation in 
organizations and facilities located outside 
the neighborhood of study. Often, research 
comes closest to understanding complex 
community changes when it relies on a 
variety of indicators, but a multi-pronged 
approach can complicate evaluation design 
and data collection. 

Using examples from their own research, 
conference participants discussed (1) key 
outcomes pursued by community initiatives 
and (2) the ways in which evaluators of 
these initiatives identified and used 
indicators of change to measure progress 
toward the desired outcomes. 
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We need to come to more precise definitions 
about family strength and what it means, 
how it’s demonstrated, how it can best be 
fostered, and how it can be measured. And 
we need to get much smarter about the 
ways in which neighborhood resources and 
characteristics can help or hinder the 
efforts of families to protect and nurture 
their kids. What kind of child and family 
outcomes are most strongly connected to 
neighborhood factors ? 

— Tony Cipollone 



Outcomes targeted by major 
interventions 

The evaluation of the Beacons Initiative, 
conducted by the Academy for Educational 
Development, sought to capture 
developmental outcomes in students who 
attended the initiative’s neighborhood 
centers, which operate at selected schools 
in New York City. (For more on the 
Beacons evaluation activities, see box on 
page 29.) The evaluation focused on 
outcomes in the initiative’s two areas of 
interest: the establishment of partnerships 
between schools, school districts, 
community-based organizations, and public 
secular agencies, and the use of these 
partnerships to bring an array of youth- 
development activities and services into the 
schools and to promote a holistic approach 
to serving young people. The evaluators 
identified the following outcomes: 

o For elementary school-aged 

children — sense of safety and physical 
well-being, sense of belonging, skills 
for resisting risky behaviors, self-worth 
and self-esteem, attachment to peers 
and adults, attachment to school, 
perceived importance of school, 
comfort in school and with school 
work, regular school attendance, high 
academic grades 



° For adolescents (in addition to the same 
outcomes listed for younger 
children) — skills in decision making, 
problem solving, and communication, 
and participation in community service 

° For adults — acquisition of new skills; 
freedom to work outside the home; 
support for children’s education; 
establishment of new friendships and 
contacts; participation in family, 
school, and community activities 

° For schools and other institutions — a 
role in organizing family-centered 
activities, support for parents’ role in 
developing their children’s educational 
and social skills, closer relationships 
between the institution and families, 
institutional support for children’s 
healthy behavior 

° For neighborhoods — actual and 

perceived areas of safety, awareness of 
and access to community resources, 
contact and relationships across 
community groups, collective problem 
solving, resident influence over the 
nature of community transformation 

Evaluators of the Rebuilding Communities 
Initiative identified five broad areas in 
which they expected to see neighborhood 
outcomes: the capacity and impact of 
neighborhood resources and institutions, the 
effectiveness of service delivery systems, 
financial investment in the communities, 
physical infrastructure (e.g., housing) and 
social infrastructure, and the capacity of 
local government. This evaluation, 
conducted by OMG Center for 
Collaborative Learning, chose those 
outcome areas because they were broad 
enough to encompass the wide variation of 
strategies and perspectives across sites, yet 
targeted enough to generate useful 
information for the sites. 
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The Cleveland Community B uilding 
Initiative, created to address poverty in 
Cleveland, embraced a holistic, asset- 
oriented approach to service integration, 
combining both place-based and people- 
based strategies and engaging neighborhood 
residents in the change process. The key to 
this intervention was the creation of village 
councils in four neighborhoods — forums 
where residents, social service agencies, 
and other stakeholders could develop 

Multi-Site Evaluation Emphasizes 
Outcomes in Common Areas 

The Rebuilding Communities Initiative 
operated in one neighborhood or 
community in each of five cities: Boston, 
Philadelphia, Washington, Denver, and 
Detroit. This capacity-building 
intervention, which began in 1994, had a 
broad mandate to stimulate collaboration 
between leading community organizations 
and other key players that would 
ultimately create, develop, or strengthen 
neighborhood governments, processes, 
and structures. Collaborators at each site 
established their own strategies for 
developing and empowering local leaders, 
supported by technical assistance. 

Evaluators used site visits to collect 
qualitative data on local context and to 
inform the development of cross-site and 
site-specific indicators of progress. 
Evaluators also administered a door-to- 
door community survey to block groups 
within each neighborhood. 

Initially, evaluators expected to focus half 
of the survey on cross-site issues and half 
on site-specific issues, to capture the 
variation of strategies across sites. 
However, they found that each site, 
although very different from its 
counterparts, related to the same core set 
of outcomes and indicators. The resulting 
survey primarily used measures common 
to all five communities. 



strategies and form links with business and 
civic leaders outside the community. 

The evaluation of this intervention, 
conducted by the Center for Urban Poverty 
and Social Change at Case Western 
Reserve University, initially targeted 
outcomes in five areas: formation and 
operation of inclusive village councils; 
identification and incorporation of 
neighborhood assets; formation of an action 
agenda for each community; development 
of high-quality program characteristics; and 
development of collaborative, city wide 
partnerships. 

In the above examples and other 
discussions of outcomes, two broad themes 
emerged: (1) the importance of measuring 
early, interim, and long-term outcomes; 
and (2) the need to distinguish individual 
results from neighborhood outcomes. 

Measuring early, interim, and long-term 
outcomes. Researchers generally accept 
the value of assessing early, interim, and 
long-term outcomes — not just the final 
outcomes. Interim outcomes are especially 
important because they reveal the year-to- 
year nuances that produce ultimate social 
change and they make explicit the links 
between actions, benchmarks, and 
outcomes. For example, by understanding 
changes in parent engagement, such as 
involvement in children’s education, 
researchers may gain an earlier and clearer 
understanding of long-term increases in 
children’s test scores. 

Valuable interim outcomes can be directly 
or indirectly related to the ultimate 
outcome. Improved school attendance and 
engagement in course work are interim 
outcomes along a direct path to improved 
learning and academic performance, for 
instance. But, as Beacons site leader Sister 
Mary Geraldine observed, if the purpose of 
a good education is to get a good job, 
development of the social competencies 
needed to obtain and hold a job — especially 
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interpersonal skills such as communication, 
conflict resolution, and collaboration — are 
equally important interim outcomes. Thus, 
evaluation frameworks need to assess many 
types of outcomes. 

Poverty Intervention Supports Four 
Diverse Pathways to Outcomes 

The Cleveland Community Building 
Initiative (CCBI) grew out of the 
Rockefeller Foundation’s Community 
Planning and Action Project, a six-city 
effort in the 1980s to galvanize 
communities to address poverty. Those 
activities in Cleveland included a study of 
neighborhood poverty by Claudia Coulton 
at Case Western Reserve University’s 
Center for Urban Poverty and Social 
Change. The report spawned a 
commission on poverty, and CCBI was 
developed to implement the commission’s 
recommendations . 

The initiative created a village council in 
each of four neighborhoods to serve as the 
change agent. Each of the four village 
councils has a strategic plan for action. 

One created a school-based family 
education center; another gained seats on a 
local development organization’s board in 
order to improve housing. The third 
council focused on improving outcomes 
for youth at risk by working with the 
juvenile justice system, and the fourth is 
developing an information and referral 
system to improve access to services. 

The CCBI evaluation used a theories of 
change approach. Evaluators involved 
residents and other community 
stakeholders in defining interim and long- 
term outcomes, benchmarks of progress, 
and measures. Data sources for 
measuring these benchmarks included 
Census data, interviews with council 
members, self-assessment questionnaires, 
and residents’ applications for council 
membership. 



Researchers continue to struggle to help 
initiatives define interim outcomes, 
however. Local change agents usually can 
describe their immediate and ultimate goals 
more easily than they can articulate the 
changes they hope to, see along the way, 
especially at the beginning of a multi-year 
initiative. 



Initial and early outcomes — these early 
events that are important in the life of [the 
initiative], and the ultimate outcomes, were 
pretty clear. [But interim outcomes were] 
really fuzzy. We’d keep pushing people 
along, asking what comes next, what comes 
after that. Somebody finally said, “Do you 
want to know what I’m going to do in Year 
5? Ask me in Year 4. ” 

— Sharon Milligan, evaluator of the 
Cleveland Community Building Initiative 



Distinguishing individual or resident 
results from neighborhood or community 
outcomes. An intervention may seek 
changes in individuals and families, in 
neighborhoods, or at both levels. This has 
two implications for social research. First, 
as noted above, evaluations must use data 
collection and analysis methods that are 
appropriate to the level of change being 
sought. Second, evaluators must weigh the 
importance of each type of outcome in 
relation to the others. 

In one sense, it isn’t an either/or 
proposition; neighborhood outcomes are 
closely related to resident outcomes. For 
example, evaluators of the Rebuilding 
Communities Initiative suggested that, since 
neighborhoods mediate the experiences of 
individuals, changes measured in 
neighborhoods could be viewed as 
intermediate outcomes along the path to 
ultimate outcomes for children and 
families. This approach allowed the 
researchers to identify outcomes that might 
not be possible to capture at the individual 
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level, especially within a timeframe of just 
a few years. Conversely, evaluators of the 
Beacons initiative documented changes in 
individuals that could suggest broader 
changes in social capital, such as an 
improved ability to collaborate across 
ethnic boundaries (although the initiative, 
which establishes youth development and 
social service programs in schools, was not 
explicitly designed as a community-building 
endeavor and is not being evaluated in 
those terms). 

Researchers do not necessarily agree on the 
way in which neighborhood conditions and 
outcomes affect individual outcomes, which 
raises concerns about using one result to 
indicate or explain the other. In addition, if 
a community transformation initiative is 
successful, some families will develop 
capacities that enable them to move to 
other, more promising communities. While 
the outcomes for these families are 
significant, changes in the neighborhood 
overall might be much less dramatic, even 
immeasurable. Unless evaluators can track 
these families, which is extremely difficult 
to do, the evaluation risks losing its ability 
to capture the real change. 

The evaluation of the Jobs-Plus 
intervention, conducted by Manpower 
Demonstration Research Corporation, 
addressed the issue of individual versus 
neighborhood effects with a dual-level 
approach. Evaluators plan to track 
individual participants from the intervention 
into the workforce to learn what proportion 
of people living in the housing 
developments are working and what their 
characteristics and work experiences are. 
The evaluation also will examine trends in 
workforce participation and the use of 
public assistance to see whether they 
change over time, and will compare them 
to trends at comparison sites to see whether 
the changes can be attributed to Jobs-Plus. 



Employment Intervention Seeks 
Changes in Both Individuals and 
Communities 

The goal of Jobs-Plus is to dramatically 
increase employment to residents of eight 
public housing developments, with an 
expectation that the lessons learned 
might apply to efforts nationwide to 
address concentrated urban poverty. 

This demonstration program, which 
operates in Baltimore, Chattanooga, 
Cleveland, Dayton, Los Angeles (two 
sites), Seattle, and St. Paul, gives 
leadership responsibility to local 
partnerships of residents, the public 
housing authority, the public assistance 
administration, job training programs, 
and community agencies. 

Jobs-Plus helps sites adopt good job 
training practices, works to ensure that 
jobs pay enough to make working 
worthwhile (which also means 
renegotiating rent policies and welfare 
rules to erase disincentives for 
employment), creates community support 
for employment, and removes barriers 
such as lack of day care or 
transportation. Sites receive technical 
assistance to develop collaborative 
relationships and design programs. 

The Jobs-Plus evaluation examines the 
implementation of these components, 
focusing on what impact the initiative has 
on work and, consequently, on quality of 
life for the residents and the community. 
Data collection includes baseline and 
follow-up surveys and interviews, 
observations, and case studies conducted 
by on-site researchers. 
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Indicators and measures used to 
evaluate the outcomes of major 
interventions 

How do evaluators identify ways to 
measure outcomes? Conference 
participants provided several examples. 

Because the Rebuilding Communities 
Initiative was intentionally broad and did 
not have clear definitions for expected 
actions, its evaluators used a theories of 
change approach to find ways of measuring 
progress, evaluator Tom Bums recalled. 
“We began with what we had, which was 
those five very broadly stated [outcome] 
areas, and we said ... ‘What are the kinds 
of things that we think could possibly 
happen over the timeframe of this 
initiative?’” 

Evaluators talked with program staff at 
each site to learn their assumptions, 
expectations, and planned pathways for 
change and to determine what information 
site leaders needed to improve their work. 
This generated a comprehensive list of 
indicators that reflected the sites’ major 
program areas. The evaluators then 
grouped the indicators into categories, 
corresponding to the program areas that 
they expected each site to pursue. They 
created a logic model, linking the strategies 
that emerged at the sites with concrete 
program areas and activities. 

“The process went on for some time with 
the program staff [and] was repeated with 
each of the five sites as they were still in 
the very early stage of shaping their plans 
and strategies,” Burns said. Each site also 
received technical assistance to develop an 
information system that would support data 
collection and analysis. 

Through this process, the evaluators 
developed the following indicators: 



° The character and capacity of 

infrastructure (i.e., social and physical 
resources) within neighborhoods 

o Lead organizations’ roles in the 

community as conveners, facilitators, 
and coordinators of community- 
building efforts — especially their 
inclusiveness, ability to collaborate, 
leadership of an agenda for change, and 
use of local assets 

o The delivery systems by which 
resources and services enter the 
neighborhood and affect families 

o The way that financial resources are 
formed (e.g., availability of mortgages, 
lending practices) 

° Residents’ perceptions of neighborhood 
institutions and resources 

o Neighborhood safety, including crime 
statistics, residents’ perceptions of 
safety, and factors that influence how 
the neighborhood receives safety 
services (e.g., number of police 
stations, officers’ training and 
experience levels) 

Evaluators of the Cleveland Community 
Building Initiative similarly began by 
involving residents, business leaders, and 
other stakeholders in developing an 
evaluation framework, explained Milligan: 

Initially, we experimented] with ... 
forward mapping. You know, asking 
' What’s your initial result, what are 
the activities that surround it leading 
to the long-term outcome, and then 
mapping backwards in terms of looking 
at the long-term ... things that 
everybody wants to accomplish. What 
actually emerged from these 
conversations [was] a generalized 
theory of change. 
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Evaluators focused on the five initial 
outcomes, described earlier, that all 
collaborators agreed were cornerstones of 
the initiative. For each target outcome, 
they talked with stakeholders about 
benchmarks that could be used to indicate 
progress and ways of measuring the 
benchmarks. For the outcome of 
inclusiveness in village councils, for 
example, the evaluators decided to examine 
the socio-demographic profile of council 
members and compare it to the profile of 
the neighborhood overall. 

The evaluations of Plain Talk, a five-site 
teen pregnancy prevention initiative, and 
Community Change for Youth 
Development, a six-site initiative designed 
to increase positive youth development 
opportunities, measured the outcomes of 
local planning and implementation. These 
evaluations, conducted by Public/Private 
Ventures, looked at the following types of 
indicators: 

• Individuals’ attitudes, behavior, and 
knowledge about relevant services 

• Access to and use of services and 
opportunities in key areas (e.g., 
adolescents’ access to adult support and 
guidance within and outside the family; 
use of out-of-school time) 

• Evidence of progress by neighborhood 
governance groups (e.g., how well they 
developed, what challenges they faced, 
what roles they planned, what 
knowledge and capacities they 
acquired) 

• Quality of activities and interactions 
(e.g., content, format, and relevance of 
workshops designed to build 
neighborhood capacity; nature of tasks 
targeted by community institutions; 
status of social networks) 



• Achievement of the interim goals that 
community groups set for themselves 

In general, conference participants agreed 
that residents and community groups should 
help shape an evaluation’s indicators and 
measures, because their decision-making 
needs will help define data needs. 

Involving community members in 
developing the evaluation also builds their 
investment in the evaluation process. 

Measuring Change 

Neighborhood transformation and child and 
family development involve complicated, 
non-linear, and sometimes controversial 
processes, interactions, and relationships — 
aspects that are notoriously difficult to 
measure using traditional evaluation 
methods. Participants raised the following 
issues related to measurement: 

Change is a dynamic process. The 
processes being measured, as well as 
the target outcomes, are constantly 
shifting. Evaluation approaches must 
be flexible to accommodate these 
changes. 

As neighborhoods strengthen systems and 
relationships, they change their goals and 
their methods for achieving them. To 
accurately measure these moving targets, 
researchers must recognize when the initial 
target outcomes are no longer relevant or 
require different measurement techniques. 
Most participants appeared to support an 
evaluation style that begins with clearly 
defined expected outcomes but measures a 
broad enough range of indicators, and 
remains flexible enough, to allow shifts in 
focus that match an initiative’s evolving 
goals. 

Although several researchers noted that the 
“fuzziness” of some interim outcomes 
results in collecting some data that later 
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prove unimportant, several participants 
echoed evaluator Robert Halpern’s 
argument that “what evaluators really need 
to do is wait as long as they possibly can 
before trying to decide what the story is ... 
and in a sense not trying to figure out what 
they’re measuring as outcomes until the 
very last moment,” because the precise 
elements that need to be measured may 
change over time. This is not to say that 
evaluators should not engage in their work 
until the intervention is well under way, 
Halpern added; rather, they should 
concentrate at first on understanding and 
documenting the early dynamics of decision 
making, priority setting, and relationship 
building among collaborators. 

The theories of change approach to 
evaluation represents a direct effort to 
accommodate evolving outcomes in an 
evaluation design. 

This approach engages stakeholders and 
evaluators in defining target outcomes, 
activities that will lead to the outcomes, and 
indicators of progress. The approach tries 
to make an initiative’s assumptions and 
goals explicit and, by placing a priority on 
measuring interim outcomes, to spot 
potential shifts in the initiative’s focus. 

The theories of change approach gets mixed 
reviews from researchers. Some 
participants suggested that these self- 
reflective qualities make the approach more 
useful to the people responsible for the 
initiative’s actions than to the evaluators 
trying to assess them. The approach does 
not necessarily provide a framework for 
understanding the dynamic process of 
linking current activities and outcomes to 
future priorities and problem-solving 
capabilities, some participants said. 
However, other evaluators described 
benefits of the theories of change approach 
for both researchers and practitioners: 



o Although it can consume a lot of 

resources, the approach helps to create 
“rigor and discipline” out of 
comprehensive change initiatives that 
have ambitious goals but no framework 
for action, said Gertrude Spilka, an 
evaluator of the Rebuilding 
Communities Initiative. “fl]t sets some 
expectations, and real beginnings, and 
some clear paths about how one might 
get there,” she said. 

° • It engages diverse stakeholder groups in 
considering how their actions relate to 
desired outcomes, and these discussions 
keep local players in the information 
loop normally reserved for evaluators 
and funders. 

° It forces evaluators to start examining 
the fit between an intervention’s goals, 
actions, and claims of success at the 
beginning of an initiative, instead of 
trying to reconstruct the pieces at a 
later date. 

® It focuses on expected outcomes and is 
open to revision and modification if the 
theories do not hold true; thus, the 
approach is flexible enough to permit 
an evaluation to change alongside an 
intervention. 

Even critics of the theories of change 
approach suggested that more front-end 
effort to define goals and pathways for 
change — and perhaps to build community 
consensus on them — would produce better- 
designed interventions and evaluations. 

The confluence of multiple initiatives 
in some communities complicates 
efforts to measure specific effects. 

An abundance of interventions and, 
consequently, evaluations within a single 
community increases the burden on 
evaluation respondents, makes it difficult to 
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establish causality, and possibly dilutes the 
impact of any single initiative. “I 
remember being in Seattle and realizing at 
one point that one resident could have a 
Jobs-Plus case manager, a case manager for 
[a federal job training program], a welfare 
case manager, a case manager for this 
refugee center . . . and who knows how 
many others,” recalled Jobs-Plus evaluator 
James Riccio. Tracing the precise effect of 
each influence is a challenge for 
researchers. 

The complicated nature of individual, 
family, and community change makes 
it especially important to measure 
change at a variety of levels, using a 
variety of methods. 

Researchers measuring social capital, for 
instance, can capture several dimensions of 
change by examining the levels of 
community engagement and trust, the 
characteristics of social networks, and the 
extent and effectiveness of community 
institutions, as Rohe suggested. To develop 
a deeper understanding of social change, 
however, evaluators should measure several 
aspects of each of these constructs. 
Community trust can be assessed not only 
by how residents perceive their 
neighborhoods but also how institutions 
perceive the neighborhoods in which they 
work and how this influences their efforts 
to engage residents. In fact, researchers 
run a smaller risk of losing valuable 
information if they analyze community 
change on a variety of separate indices, 
such as the constructs of social capital 
outlined above, than if they try to create a 
composite scale of social capital by 
blending results on the component 
dimensions. 

Multi-measurement approaches are 
especially important because neighborhoods 
and families are interrelated. Each 
dimension of family and community life 
affects outcomes for individuals and 



neighborhoods. As one participant noted, 
“It’s not just because social capital is high 
that everything’s going to go well.... In 
every community you have to see what the 
effects of various elements are, and how 
each one plays a part. You have to get at 
what the strengths and weaknesses are and 
how they affect each other.” 

Multiple-measurement approaches 
promoted by participants include: assessing 
residents’ perceptions, experiences, and 
characteristics through surveys; collecting 
demographic and process information from 
administrative and statistical data bases; 
gathering contextual data through 
interviews and observations; and 
developing case studies and ethnographies. 

An intervention may appear to fail not 
because it was poorly designed or 
implemented but because it had 
insufficient resources to meet its 
goals. Evaluations should measure 
outcomes that can reasonably be 
produced given the available 
resources. 

Lack of resources may affect the kind of 
outcomes that are achieved and, 
consequently, measured by evaluations. 

For example, an intervention that has 
inadequate human or financial resources 
may end up doing a little good for a lot of 
people, when its purpose — and therefore 
the yardstick by which its success was 
measured — was to do a lot more good for 
fewer people. “One way a lot of 
evaluations of these kinds of initiatives are 
set up to show failure is if the initiative 
defines outcomes at ... a broad level [but] 
has nowhere the magnitude of resources to 
impact in that amount of time,” observed 
researcher Rick Brandon. 

When evaluators are deciding which 
outcomes to measure, Brandon added, they 
should consult with local stakeholders about 
how much progress toward each goal can 
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reasonably be expected given existing 
resources. 

Evaluations do not always consider 
the effect of ecological factors on the 
outcomes they measure. These 
factors include environmental risks 
and community conflict or 
controversy. 

During a discussion of the Rebuilding 
Communities Initiative, conference panelist 
James Johnson urged evaluators to think not 
only about obvious indicators, such as 
neighborhood crime and unemployment, 
but also the imperceptible factors that pose 
developmental risks for urban children and 
families, such as exposure to lead-based 
paint in housing developments and 
neighborhood institutions. “To what extent 
are those issues factored in, when we know 
these things affect learning and child 
development, family development?” he 
asked. Finding a home or a child care 
program for a family might be considered 
an asset, he added, but “when you look at 
the developmental risk when you put a kid 
in an environment that’s contaminated, you 
may be doing him more harm.” 

Although competing priorities usually push 
environmental concerns to the bottom of the 
list for initiatives and their evaluations, 
Johnson suggested that evaluators should at 
least ask local stakeholders about their 
. awareness of and concern for those factors 
that are not easily perceived. 

Similarly, research rarely captures the 
shape and effects of neighborhood 
controversies, despite their prevalence in 
the communities where interventions 
typically occur. “As attractive as the 
concept [of social capital] is, what strikes 
me is that it leaves out realities of social 
conflict [and] competition,” observed Sister 
Mary Paul Janchill, director of a Beacons 
site. “Sometimes you can have too much 
[emphasis on] cohesiveness, agreement, 



collaboration. I’m not sure that it does 
justice to some of the real challenges.” 

Researchers, evaluators, and change agents 
all acknowledge that community conflict or 
controversy affects the process of social 
change and the achievement of outcomes, 
but none of the parties have paid adequate 
attention to understanding how, agreed 
Garland Yates, a senior associate at The 
Annie E. Casey Foundation. “It is so 
difficult to evaluate and assess [that] it falls 
off the table at some point,” he said. “We 
have to embrace the challenge to figure that 
out.” 



Because measuring community conflict is 
difficult, "we have the struggle we have in 
terms of how to get people to talk about 
social networks and the development of 
relationships beyond [the fact] that they are 
good things to do. There ought to be a 
rationale and a reason to them, and ... the 
reason is about the genesis of conflict and 
controversy. ” 

— Garland Yates 



Conductine Comparative Research 

Several issues make it difficult to conduct 
comparative research on complex, 
comprehensive initiatives: 

e> Every neighborhood is subject to 
multiple influences and interventions. 
Regional economies and programs 
unrelated to the community-building 
initiative, such as welfare reform or 
rent incentives for public housing, can 
influence outcomes in ways that make it 
difficult to attribute differences to a 
particular intervention. 

° Service providers who operate in 
both intervention and comparison 
sites may change their services in the 
comparison site as a result of 
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participating in the intervention site. 
This change in practices blurs the 
distinction between the two sites. 

• Neighborhoods may not have 
demographic and contextual 
characteristics that are similar 
enough to support comparisons. 



You can ’t assume that the control group is 
a no-treatment group .... [T]he burden for 
the evaluation is to measure that and say, 
well, they had some extra treatment but 
how much difference in treatment is there 
and of what kind? 

— James Riccio 



Despite these challenges, several 
approaches allow researchers to make 
comparisons. One technique is to carefully 
monitor services in a comparison site so 
that, if there is a finding of “no impact,” 
evaluators can determine whether the 
intervention failed or whether the 
comparison site managed to deliver 
comparable services without the 
intervention. For example, the Beacons 
evaluation surveyed middle-school students 
on selected outcomes in Beacons schools 
and in non-intervention schools that are not 
within walking distance. The surveys 
included questions about after-school 
activities so evaluators could rule out 
similar activities as a possible explanation 
for similar outcomes. 

Comparative research on community 
change can also be conducted by randomly 
assigning whole communities, rather than 
individuals, to treatment and control 
groups. This is the case in the Jobs-Plus 
evaluation, which uses entire housing 
developments instead of individual residents 
as its sample points. The Jobs -Plus 
initiative, which concentrated employment 
interventions in public housing 
developments, required cities applying for 
the demonstration grants to identify three 



developments that met the grant criteria. 
From each of these clusters, Jobs-Plus 
selected one site for funding and used the 
other two as comparison sites. Evaluators 
conducted baseline surveys and are 
monitoring changes in these sites as well as 
the sites involved in the actual Jobs-Plus 
intervention. 



Timing Evaluation Activities 

Two timing issues shape the focus and 
nature of evaluation: (1) the point in an 
intervention’s life at which an evaluation 
begins, and (2) the stages that an evaluation 
moves through over the course of an 
initiative. 

When should evaluation begin? 

Until recently, evaluations rarely started at 
the same time as the interventions they 
sought to explain. It could be years or even 
decades before researchers formally 
assessed outcomes, during which time the 
intervention had grown, shifted in focus, or 
sometimes disappeared. This norm is 
changing as funders’ expectations for early 
accountability grow and as practitioners see 
the need for earlier feedback on their 
efforts. Researchers still have mixed 
opinions about the optimum time to begin 
an evaluation, however. 

When evaluators wait to begin measuring 
change until the initiative is well under 
way, neighborhood collaborators gain time 
to sort out their new roles and 
relationships, balance the competing 
interests of various funders, and build trust 
and ownership among partners. But the 
decision to delay evaluation until after the 
initiative’s planning and start-up phases are 
completed makes it harder to capture 
baseline conditions and important early v 
events, interactions, and processes. 
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Documentation — research that records the 
story of what is happening, without 
necessarily evaluating impact or 
effectiveness — offers a way to collect 
important early information while 
researchers wait to define and design their 
evaluation. The purpose of documentation 
is to capture descriptive data on an 
intervention’s emerging processes, 
activities, interactions, and patterns of 
behavior to establish a reference point for 
later analysis and reflection. 

Documentation not only creates an 
historical record of an intervention, it may 
also make qualitative data available long 
before researchers have agreed on the 
evaluation’s purpose. 

How does timing affect evaluation 

activities and interests? 

Research and evaluation are iterative 
processes, much like interventions 
themselves. As researchers refine their 
understanding of an initiative, as the 
initiative itself unfolds, and as data needs 
change, the research focus evolves. 

Participants generally agreed that early 
research activities include clarifying the 
goals of the initiative and the evaluation, 
documenting the process of change, 
developing appropriate indicators of 
change, and establishing a baseline for data 
that can be tracked during the life of the 
intervention. Through these endeavors, 
researchers concentrate on learning whether 
their initial presumptions about the 
intervention hold true and whether the 
intervention’s own short-term objectives are 
being met. For instance, are collaborative 
groups being formed in the neighborhood, 
and are they developing viable partnerships 
that can strengthen families and promote 
community change? What evidence exists 
that collaboration actually is occurring? 

The process of developing indicators of 
outcomes usually begins during this first 



stage of evaluation design, and if 
stakeholders from the intervention 
participate, it can be as useful for the 
intervention as it is for the evaluation. 

These early indicators will not necessarily 
last the duration of the evaluation, 
participants cautioned; they will be refined, 
discarded, and supplemented as the 
initiative takes shape. Also, during the 
early stages of implementation an initiative 
may be too amorphous to support accurate 
measurement against the indicators. 
“[Ijndicator work happens progressively,” 
observed Tom Burns, an evaluator of the 
Rebuilding Communities Initiative. “It 
messes us up if we get too caught up in a 
reliance on indicators at early phases of 
these initiatives ... and pretend there is a 
precision and a rigor at moments in these 
initiatives when it just isn’t there.” 

Research that tries to monitor early 
developments on a fast timeline — for 
example, on a monthly basis — runs a risk 
of over-analyzing the few actions that occur 
early on, one participant suggested. 

Slowing the pace of analysis to a semi- 
annual timeframe may yield more 
significant findings of change. 

As the intervention takes hold, researchers 
continue to refine their analyses of the 
change process and also begin to assess 
program development and quality, 
outcomes, and effectiveness. Eventually, 
researchers should begin to provide 
feedback to local collaborators and build 
local capacity for self-evaluation and data 
use. In most traditional evaluations, these 
activities occur fairly late in the research 
schedule, if at all. Participants urged their 
colleagues to share data and analysis with 
sites more frequently, in time to help 
inform strategic action and before the 
findings become dated. Issues related to 
feedback and local data use are discussed in 
more detail on pages 38-43 of this 
summary. 
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Integrating Qualitative and Quantitative 
Data 

Quantitative data, from surveys and 
analyses of statistical or administrative 
databases, and qualitative data, from 
interviews, focus groups, structured 
observations, and case studies, reveal 
different aspects of the same complex story. 
By including both types of data in their 
work, researchers can describe family and 
community change more deeply, 
accurately, credibly, and compellingly. 

For example, the quantitative evaluation of 
the Jobs-Plus initiative uses surveys and 
administrative data to examine employment 
outcomes related to institutional change and 
services and to assess the feasibility of 
operating the intervention in diverse 
settings. A qualitative component, led by 
on-site anthropologists, incorporates 
observation, informal interviews, and case 
studies of the experiences of individuals 
who participate in the initiative. The 
evaluation of Plain Talk supplemented 
baseline and follow-up surveys with 
ethnographic research at three in-depth 
study sites. On-site researchers interviewed 
local participants and observed activities, 
focusing on the nature and quality of 
interactions among community members 
and between the community and the 
intervention. 

Qualitative and quantitative data should be 
integrated and mutually reinforcing so that 
each augments the other. Thus, for 
example, evaluators of the Beacons 
initiative use results from site observations 
to guide instrument development for 
surveys, interview people after they have 
completed surveys, and enter qualitative 
data into databases that support tabulation. 

Participants raised two concerns about 
qualitative research in evaluations of 
complex social change: 



• The quality of data collection can 
vary widely, especially in large-scale 
evaluations that rely on many observers 
and interviewers and in two-tiered 
evaluations that involve local 
participants in data collection. Key 
gaps can occur in (1) site visitors’ 
knowledge about the intervention, 
framework for understanding what they 
see and hear, and skills in interviewing 
and understanding; and (2) community 
representatives’ capacity to describe 
their philosophies and strategies. 

• All members of a research team may 
not reach the same conclusions from 
qualitative data, or the data may fail 
to support impressions derived from 
observations. Researchers bring their 
own views and experiences to an 
interview or site visit, and these 
individual characteristics color their 
perceptions. Also, because site visits 
allow researchers to observe activities 
and factors beyond the scope of the 
evaluation, site visitors may reach 
conclusions that cannot be supported by 
the data. As a result, some of these 
conclusions cannot be reported. “We 
saw really interesting youth 
development activities — leadership 
groups, community service — but it 
wasn’t clear to us how many kids were 
actually participating in those high- 
intensity, high-quality activities. 

Nothing in the way we had actually 
structured the site visits enabled us to 
really get at that issue,” one evaluator 
acknowledged. 

Evaluators of the initiatives discussed at the 
conference addressed these concerns by 
providing extensive training and 
opportunities for discussion among team 
members. 
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Combining Qualitative and 
Quantitative Research 

The Beacons evaluation encompassed: 

❖ Visits to Beacons sites, where 
evaluators interviewed intervention 
leaders, practitioners, community 
members, and participants 

❖ Observation of youth development 
activities, using a structured 
protocol 

❖ Focus groups with parents of 
children at Beacons schools 

❖ Structured debriefings of site 
observers to identify emerging 
patterns and gaps in collected data 

❖ Creation of a computerized 
inventory of qualitative data, with 
items based on site observers’ 
answers to 200 limited-response 
questions 

❖ Interviews of long-time participants 

❖ Focus groups with site 
representatives to learn about new 
developments, outcomes, 
challenges, and solutions and to 
engage these people in helping to 
rate the quality of specific activities 

❖ “Snapshot surveys” of 7,500 
students that collected basic 
demographic data and information 
on duration and frequency of 
participation; students who filled 
out the survey were invited to 
participate in short interviews 

❖ Surveys of participants (often 
paired with interviews), parents, 
school faculty, and other staff 



Using a Two-Tiered Approach to 
Combine National and Local Evaluations 

Increasingly, researchers, program funders, 
and community collaborators are 
recognizing the value of research that links 
evaluation on a national scale with 
individual, site- specific evaluation. Often, 
the local evaluations are conducted by or 
with assistance from community 
representatives, making them opportunities 
to build local capacity for self-assessment 
and reflection. National evaluations also 
benefit from this two-tiered approach. The 
local evaluation can become an information 
conduit, enabling the national study team to 
collect certain data and to share important 
cross-site data with each community. 

There are many models for two-tiered 
evaluations. In some cases, the national 
and local evaluations serve separate 
purposes. Although the two activities may 
include communication and information 
sharing across studies, they operate 
independently and have different goals. In 
the Neighborhood and Family Initiative 
evaluation, for example, the national 
evaluation is a broad study of the 
usefulness, viability, and implementation of 
the initiative’s two guiding principles. The 
local evaluations are autonomous and 
locally governed; their purpose is to track 
outcomes and progress toward goals 
identified by the sites and to give the sites 
feedback on their achievements. 

In other evaluations, the national study is 
the framework for all research and the local 
evaluations are primarily components of the 
larger evaluation. Evaluators of Jobs-Plus, 
for example, described their study as “a 
national evaluation in seven sites,” 
supported in each community by local 
evaluation leaders who are employees of 
the national evaluation team. 
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Two-Tiered Evaluation Model 

The Neighborhood and Family 
Initiative (NFI) sought comprehensive 
responses to the needs of children, 
families, and neighborhoods, grounded 
in community collaboration and 
participation. It operated through 
community foundations in Detroit, 
Hartford, Memphis, and Milwaukee. 
The national evaluation was a broad 
study of the implementation process, 
strategies, and structures. It focused on 
the usefulness and viability of NFI’s 
guiding principles for change. The 
autonomous local evaluations, governed 
by local providers, tracked outcomes 
related to site-specific objectives and 
provided feedback to the community. 



Participants described the following 
challenges to designing and implementing 
two-tiered evaluations: 

Establishing complementary purposes, 
roles, and relationships for national 
and local evaluations 

The variety of possible approaches for two- 
tiered evaluations makes it essential that 
national and local evaluators clearly define 
their respective goals and work out a 
relationship that supports both levels of 
evaluation. In particular, the funder, 
national and local evaluators, and site-level 
leaders of the intervention must understand 
“who’s doing what, who’s responsible for 
what, who is liable to whom, and how the 
information will be shared with the 
multitude of audiences,” summarized one 
evaluator. 

Attempts to clarify evaluation roles and 
objectives are influenced by: 



The nature of the initiative itself. For 
example, evaluators of the 
Neighborhood and Family Initiative 
designed the local evaluations as 
autonomous studies because the very 
broad principles of the initiative, and 
the extreme differences in local 
context, left the national evaluation 
without a single theory of change to 
track across sites. 

Expectations for what local data 
analysis might achieve. Establishing 
local responsibility for evaluation, 
instead of centralizing control of 
research at the national level, can push 
sites to develop skills in strategic 
planning, goal setting, data collection, 
and information feedback and to make 
an investment in self-evaluation that 
outlasts the national evaluation. In 
some cases, this is an explicit objective 
of the intervention’s funder. 

Varying definitions of what 
constitutes evaluation. As one 
participant observed, the word 
“evaluation” is used loosely to mean 
many things: analyses of outcomes 
against strict performance criteria, 
broad studies of implementation 
processes, assessment activities 
intended to create opportunities for 
learning and reflection, and assessment 
of compliance with the specifications of 
a grant. This can cause confusion and 
unrealistic expectations between local 
and national evaluations. 

The information needs and interests 
of local and national stakeholders. 

Intervention sites may need evaluation 
data for program management and 
accountability, while national 
evaluators want data to inform broader 
cross-site findings about social change. 
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It was not pretty at times — trying to make 
sure the site perspective was there, making 
sure the [research organization ’s] 
perspective was there, the foundation ’s 
perspective ... and all of us saying. Okay, 
we have these critical elements. We have a 
place we want to go. We have things we 
want to demonstrate. The sites have their 
perspective; we have ours. Now how do we 
hammer this out? 

— Sherece West, Program Associate to the 
Rebuilding Communities Initiative, 
The Annie E. Casey Foundation 



Capturing both shared and unique 
elements of multi-site initiatives 

The stories of individual communities that 
produce positive outcomes for children and 
families are extremely valuable. They can 
be informative and inspirational. They 
capture the imagination of funders and 
policy makers, build momentum for local 
action, and spark debate about specific 
issues. But they don’t necessarily capture 
the common elements that are essential to 
all projects within an initiative or produce 
cross-cutting lessons that apply to other 
neighborhoods and contexts. 

Two-tiered evaluations have an opportunity 
to try to understand both the common 
elements and the crucial differences among 
an initiative’s sites. For example, the local 
evaluation component of the Neighborhood 
and Family Initiative tracked outcomes 
related to local objectives. Although the 
outcomes varied substantially across sites, 
the national evaluation component looked 
across sites and identified a theme: While 
the notion of comprehensive change 
spurred local collaboratives to think 
broadly, it did not provide sufficient 
guidance to help sites do so, often resulting 
in community-building responses that 
addressed needs more categorically than 
comprehensively. From this observation, 



national evaluators could extract lessons 
about the guiding role of funders, the 
capacity of local collaborators to think 
strategically, and the effects of unequal 
power among local collaborators. 

It’s often hard to make cross-site 
comparisons, however. If local evaluations 
are free to pursue their own research 
agendas, they may not produce much 
comparable information. The extreme 
diversity of initiatives that tailor change to 
local context can further limit the 
applicability of lessons from one site to 
another. Also, the depth and quality of 
data collection frequently is uneven across 
sites. Some sites are more willing or able 
than others to share relevant, in-depth 
information, so it is always possible that a 
cross-site research finding could have been 
supplemented or contradicted by data that 
evaluators could not elicit from an 
unresponsive site — or that experiences at an 
extra-cooperative site disproportionately 
influenced cross-site findings. 



We’re trying to take advantage of the fact 
we have seven places trying to do 
something similar but they may do it in 
different ways, and what can we learn from 
that. At the same time ...we want to 
capture some of the [site-specific change] 
as well. 

— Evaluator of the Jobs-Plus initiative 



Timing the national and local levels 

of study. 

Local and national evaluations are more 
likely to produce compatible, mutually 
useful data if their areas of focus and 
research activities are integrated in some 
way. This does not mean that every aspect 
of the two tiers must follow identical paths, 
but there should be some shared ground and 
some process for communicating. 
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Timing plays a significant role in 
integrating national and local evaluations. 
Local evaluations that are initiated long 
after a national evaluation begins and 
developed entirely apart from the cross-site 
research may struggle to gain a place in the 
data collection, analysis, and feedback loop 
of the national study. This was the case for 
the local evaluations of the Neighborhood 
and Family Initiative. “I’d describe the 
basic problem as compartmentalization of 
duties and concerns that never quite 
merged,” recalled evaluator Robert 
Chaskin. “There was some level of distrust 
... at the local level in terms of what 
evaluation is about and why we should do 
it, and lack of clarity about what the 
national cross-site evaluation was to do.” 

Timing can also influence the allocation of 
resources to each tier of research. For 
example, it may make sense to gradually 
shift some resources from the centralized 
national evaluation to the local assessments, 
as local activities and effects expand over 
time, one evaluator suggested. 

Building local trust, support, and 

capacity for evaluation 

Communities often resist evaluation. Their 
resistance usually is driven by several 
factors: (1) concern that evaluators will try 
prematurely to find evidence of transformed 
families or communities; (2) fear of 
negative consequences; (3) distrust of 
evaluators and a belief that the evaluators 
can directly affect program funding; (4) 
lack of interest in evaluation, often 
exacerbated by a lack of funding for local 
evaluation; (5) lack of faith that the 
intervention’s broad and ambitious goals 
can be linked to measurable outcomes; or 
(6) lack of experience or capacity in self- 
evaluation. 

In self-defense, local collaborators often try 
to control the research. In the Plain Talk 
evaluation, for example, New Orleans site 



representatives initially thought that 
evaluators had the power to expand or cut 
off the program’s funding source. “In the 
site visits, we were always careful to put 
our best foot forward. You know — pick 
the stories we want to tell, pick the stories 
we don’t want them to know anything 
about,” admitted site director Tammi 
Fleming. Until the site overcame this 
reaction, “we didn’t really get someone to 
objectively observe our mistakes [so] we 
could change them in the long or short run 
to make the program better,” she said. 

Ethnographers and other researchers who 
spend a lot of time on site can help allay 
community fears that research will only 
reflect the quantifiable effects of change. 
But the sense that an outsider will be able 
to see “our dirty laundry,” as Fleming 
described it, still makes communities worry 
about evaluators who are total strangers to 
the community and its culture. Participants 
suggested the following tactics for 
overcoming suspicion and building local 
support: 

• Give sites some input into the selection 
of the researchers who work most 
closely with the community. This 
ensures that the researchers’ 
assumptions about poor communities, 
about evaluation, and about the role of 
evaluators in communities are sensitive 
to the neighborhood in question. 

• Set clear, reasonable guidelines for 
which aspects of the evaluation the 
community can and cannot influence. 
For instance, community members 
should be able to provide feedback on 
evaluation reports but not dictate the 
interpretation of data. 

• Position the evaluation as a central 
component of the intervention, not as 
something abstract and separate. 
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o Present evaluation as a tool all 

stakeholders can use to improve their 
work, not as a punitive or judgmental 
function. In part, this means 
understanding and explaining 
community members’ perceptions about 
why things are a certain way, not just 
describing successes and failures. 



There’s a tremendous amount of ambiguity 
about goals and expectations ... built into 
the collaboratives themselves. There was 
also a lack of interest in evaluation on the 
part of the community foundations, which 
didn ’t have much experience with it and 
pretty much adhered to a philosophy of, 

‘We ’ll know [success] when we see it. ’ 

— Evaluator of a multi-site initiative 



Technical Assistance Issues 

Researchers and funders are beginning to 
address many of the issues described in this 
summary by making technical assistance a 
vital part of evaluations. Assistance to sites 
may focus on ways to understand and 
accommodate the evaluation process, how 
to conduct self-evaluations, how to use 
evaluation data effectively, or all three 
purposes. This section describes major 
technical assistance issues identified by 
conference participants. The related issues 
of providing feedback to communities and 
building local capacity to value data are 
discussed in a later section. 

As more researchers adopt a second role of 
helping local collaborators participate in 
evaluations (or at least accommodate 
technical assistance in their studies), the 
hard line between evaluation and technical 
assistance has blurred. The experiences of 
a few evaluations with technical assistance 
have shown that researchers can play 
multiple roles in relation to a community 



and that the benefits of providing assistance 
outweigh concerns about tainting 
researchers’ objectivity. The benefits can 
include stronger relationships among 
collaborators and the introduction of an 
independent perspective on the evaluation 
process. 

Participants generally agreed that most 
evaluations benefit from incorporating some 
form of technical assistance, although the 
nature and extent of assistance depend on 
local needs, available funding, the scope 
and depth of the evaluation, and the goals 
of the intervention. In some initiatives, 
assistance is provided by the evaluators; in 
others, a separate group of assistance 
providers works independently of the 
evaluation. In both cases, the following 
issues present challenges: 

Aligning the assistance with the 

purpose of the initiative 

Technical assistance is most effective when 
it provides concrete methods for addressing 
specific needs. It is most useful to sites 
when those methods and needs reflect the 
priorities of the comprehensive initiative. 
Unfortunately, at the beginning of a new 
initiative, local collaborators, technical 
assistance providers, evaluators, and even 
funders may not know exactly what those 
priorities are. In this early stage, technical 
assistance needs to engage stakeholders in 
discussions that help them convert a vague 
sense of direction into a clear, unified 
purpose. 



If there was a clearer sense of what the 
sites wanted to accomplish, the whole 
technical assistance strategy — the whole 
issue of ownership — could have been dealt 
with more easily. 

— Technical assistance provider 
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unique interests and needs of the four sites, 
so it was replaced by site-controlled funds 
for targeted assistance. 



Differentiating between the goals, 
strategies, and expectations for 
technical assistance held by various 
stakeholders 



Although funders, technical assistance 
providers, and sites rarely discuss with 
each other their motives for engaging in 
technical assistance, each player has an 
interest in what the assistance should 
accomplish. Differences or conflicts 
among these expectations can add to the 
confusion surrounding a new intervention. 

A foundation’s primary interests in 
technical assistance are to improve the 
quality of program implementation and 
evaluation and to ensure that attempts to 
produce complex, difficult change get every 
change they need to succeed. Sites usually 
want help developing their own capacity to 
implement or assess an intervention, goals 
that are related but not limited to improving 
overall program quality. Evaluators who 
provide technical assistance want to 
improve data collection and data quality, 
which will not necessarily meet the sties’ 
goals of building local capacity for 
evaluation, although it can. And 
professional technical assistance providers 
have an interest in successfully completing 
the work for which they have been 
hired — not necessarily in thinking about 
comprehensive change in the same ways 
envisioned by the sites and funders. 

Site leaders and technical assistance 
providers may not always understand or be 
able to meet each others’ expectations. For 
example, the Jobs-Plus intervention initially 
offered assistance to all of its sites through 
a centralized team of providers. When this 
approach failed to give the depth of 
assistance that the communities wanted, the 
initiative switched to a network of local, 
part-time providers who work intensively 
on site. Similarly, the Neighborhood and 
Family Initiative’s early efforts to provide 
centralized assistance did not meet the 



Technical Assistance in Action: 

The Neighborhood Partners Initiative 

Technical assistance helped sites in the 
Neighborhood Partners Initiative collect 
data for strategic planning and goal 
setting, develop target outcomes and 
measures of success, and document early 
implementation activities. The assistance 
providers relied on four techniques: 

❖ Site-initiated assistance, with topics 
and formats selected and managed by 
each site in order to increase local 
ownership of assistance activities. 

❖ Foundation-initiated assistance on 
pre-selected strategies and activities 
that supported the funder’s interests in 
documentation and institutional 
capacity building. Although this 
assistance was not completely 
prescribed, it was strongly influenced 
by the foundation’s overall goals. 
Some standard assistance was provide 
to all sites. 

❖ Cross-site learning initiated by sites, 
which allowed sites to define the 
learning opportunities they wanted to 
have with colleagues in other sites. 
The purpose was to challenge sites to 
take ownership of the initiative. 

❖ Site participation in managing and 
coordinating the initiative. A group 
of site representatives shared 
responsibility for making decisions 
about the technical assistance strategy, 
evaluation, and other aspects of the 
initiative. 
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Using technical assistance to build 
local capacity for change 

Good technical assistance doesn’t just tell 
communities what to do or do it for them, it 
prepares community members to make their 
own plans, decisions, and actions. Instead 
of creating dependence on outside help, 
technical assistance should help local 
collaborators recognize when they need 
help, know where to get it — and, 
ultimately, outgrow the need for it. 

Examples described at the conference 
include the following: 

° Assistance to the Beacons sites, 
provided by Youth Development 
Institute, is designed to foster a 
learning community. Every month, the 
institute convenes Beacons leaders to 
discuss key issues. The Fund for the 
City of New York also links site 
leaders and staff to specialized training, 
either by conducting sessions or by 
making other opportunities available. 

® In the Rebuilding Communities 

Initiative, Metis Associates guided sites 
through the process of hiring survey 
coordinators and administrators, 
created a training manual, and trained 
survey staff. The five sites conducted 
their own surveys with ongoing support 
from Metis. 

According to conference participants, much 
more capacity-building assistance is needed 
for sites participating in complex 
interventions, especially those with local 
evaluations. 

Technical assistance shares an objective 
with the genre of evaluation that recognizes 
the value of engaging communities in skill 
building. However, assistance also 
introduces a confounding variable for 
research, because if the assistance really 



does build local capacity it affects the 
processes and characteristics being 
evaluated. 



We have totally redefined technical 
assistance. We start with very specific 
problems, so that our credibility and trust is 
built on our capacity to produce very 
tangible things that help [sites], based on 
what they see as their need — not [where] 
we think they need to be five years from 
now. 

— Technical assistance provider 



The capacity-building issue is not limited to 
sites, participants noted. Technical 
assistance providers also should be able to 
improve their skills and knowledge over the 
course of an initiative so they can adapt 
their techniques and strategies to changing 
local needs for assistance. 

Timing activities so that the assistance 

agenda matches local needs and 

interests 

The beginning of a new social intervention 
seems like a natural time to begin advising 
sites on how to produce the changes they 
seek. But communities aren’t always eager 
for help when they are deep in the throes of 
creating new partnerships, building 
relationships, and taking their first steps 
toward change. Local collaborators may 
not feel much ownership of the initiative, 
are unsure of their own roles, do not know 
exactly what assistance they need, and may 
feel threatened by outside experts. “Until 
[site leaders] had an understanding and 
some measure of ownership over the 
initiative . . . they were not going to step up 
and participate in a meaningful way in 
shaping a technical assistance strategy,” 
recalled Bill Traynor, who provided 
assistance in the Neighborhood Partners 
Initiative. 
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Early technical assistance should be simple 
and concrete; it should be driven by the 
demands of the initiative and the sites’ 
interests. As the initiative matures into a 
more complex undertaking, as collaborators 
begin to understand what specific help they 
need, and as sites accept ownership of the 
initiative, assistance can expand to meet 
new needs. 

The Neighborhood Partnership Initiative 
learned this lesson the hard way. The 
initial assistance plan was “very complex 
and overbuilt,” Traynor acknowledged, and 
“the technical assistance team was seen as 
getting in the way of the initiative.” The 
requirements for coordination and 
communication were enormous, and the 
demand for sites’ time and attention was 
excessive. At the end of the first year, an 
independent assessment of the assistance 
effort helped the collaborators reconsider 
their approach and devise a new approach 
that focused more directly on the sites’ 
specific, first-stage needs. Technical 
assistance providers stopped trying to 
analyze and fine-tune everything the groups 
were doing on a month-to-month basis and 
gave them more space to experiment with 
their work. The technical assistance 
providers and local collaborators worked 
together to define a technical assistance 
strategy that met the initiative’s own 
interests. 

“We spent a lot of time talking about ... the 
self-interest of the foundation, the self- 
interest of the sites, what’s really the 
purpose of the initiative,” Traynor said. 

The result of these conversations was a 
four-pronged approach that included site- 
initiated technical assistance, assistance on 
topics initiated by the foundation, cross-site 
learning driven by the sites, and site 
participation in managing and coordinating 
the initiative. 



Clarifying the roles of national 
evaluators and technical assistance 
providers 

Although the distinction between evaluation 
and technical assistance has softened, in 
reality it takes some effort to blend these 
two endeavors. In the early stages of the 
Casey Foundation’s Mental Health 
Initiative for Urban Children, for example, 
the evaluation and technical assistance were 
completely separate from each other. “The 
right hand and the left hand were really not 
coordinating the work. Both [parties] had 
to change the way we were doing 
business,” explained evaluator Marcela 
Gutierrez-Mavka. “The technical 
assistance decided that they couldn’t 
continue to push their expertise on the sites 
unless they were ready for it.” But what 
made them ready? That’s where the 
evaluation came in. 

“We began to look at the results of the 
evaluation ... dealing with very concrete 
issues of implementation,” Gutierrez- 
Mayka said. The evaluators replaced long, 
academic reports with short, concrete 
analyses of strengths and opportunities for 
improvement. These became the basis for 
technical assistance strategies. 

Collaborators in the sites responded well to 
the changes, because the new roles 
provided a clear path to self-improvement. 
“Four months after this report was 
produced and the training was delivered, 
we are seeing already the change in 
practice — a change that we’re going to go 
back and evaluate and give more feedback 
on,” Gutierrez-Mayka reported. 

Complementary research and technical 
assistance roles are characterized by: 

• Communication between researchers 
and assistance providers 
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Unified and seamless data collection 
processes and instruments 

° Efforts to avoid confusing sites with 
mixed messages and conflicting 
demands 

o Recognition that assistance is not the 
same as evaluation — even when both 
roles are played by the same 
people — and that assistance providers 
are not undercover grant monitors or 
“snoops” for an evaluation 

° Recognition that sites have the 

authority to help determine technical 
assistance strategies, topics, and 
activities 

Building trust and collaborative 
relationships among assistance 
providers and recipients 

At the beginning of an intervention, the 
relationships between sites, funders, 
assistance providers, and evaluators are 
tenuous. People do not know each other 
well enough to trust their partners, share 
ownership, or feel comfortable risking new 
roles. Sites typically view the foundation 
supporting the initiative as a source of 
funding, not necessarily as a partner, and 
the assistance team as an extension of the 
funder’s will and interest — or as an agent 
for the evaluation. For its part, the funder 
may view intervention sites as communities 
that need to be pushed and prodded into 
something more than they are. Efforts to 
dispel these perceptions can actually 
exacerbate the problem by making sites 
suspicious of the motives. 

Without trust, it is hard to develop a 
learning environment. Instead, the 
technical assistance team and the content of 
assistance can become the arena where 
stakeholders fight out their concerns. 



Key ingredients for trust include: 

° Time for collaborators to get to know 
each other 

o A shared sense of ownership for 
crucial research activities, so that 
neither sites nor assistance providers 
feel it is being forced on them 

o Opportunities for stakeholders to talk 
explicitly about the role and quality of 
technical assistance and evaluation, 
which allows them to recognize each 
other’s vulnerabilities and self-interests 



Implicitly, by saying assistance is needed, 
you are saying something ’s wrong, and you 
can 't say something 's wrong without 
criticizing. jTjhat puts people at an uneven 
moment in their relationship. What 
happens in those situations is that people 
revert to certain kinds of stereotypical 
behavior ... and they mobilize their 
organizations to act in organizationally 
stereotypical ways. The foundations 
become a bit more arrogant, and the CBOs 
become a bit more defensive. 

— David Hunter, Director of Assessments 
Edna McConnell Clark Foundation 



Foundations have a major role in 
supporting activities that build trust. In 
addition to creating opportunities for 
dialogue among stakeholders, they can 
demonstrate that they are prepared to do 
some of the hard work themselves — and 
that they are eager partners in change, not 
just monolithic institutions that can 
command sites and assistance providers to 
work together. For example, foundations 
can show flexibility in timelines, delegate 
more authority to local collaborators, and 
change their internal procedures so that 
promised resources become available more 
quickly. These kinds of changes helped the 
Edna McConnell Clark Foundation build 
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trust among local collaborators and 
assistance providers involved in the 
Neighborhood Partners Initiative. 

Presenting Evaluation Data in 
Useful and Compelling Ways 

The best efforts to measure change and 
assist sites with research will not 
accomplish much if the data don’t reach the 
people charged with promoting, shaping, 
and implementing change in a format they 
understand and appreciate. Conference 
participants discussed this issue in the 
context of documenting the process of 
change, providing communities with 
feedback, promoting data as a tool for local 
advocacy, and building local capacity to 
value and use data. 



Shared Ownership is Vital 

Assistance providers speculate that the 
Neighborhood Partners Initiative’s effort 
to document the change process faltered 
because neither the assistance providers 
nor the sites were invested in the 
activity. The documentation work asked 
local leaders to complete information 
logs and meet frequently with 
researchers to reflect on their logs and to 
map the course of the initiative as it 
evolved. Unsure of how the 
documentation related to the technical 
assistance and to the foundation, and 
therefore uncertain why they should 
commit to this time-consuming activity, 
community leaders were leery of 
speaking candidly with documenters. 
Although the project did yield some 
useful information, “We never really 
were able to create a sense of honest 
dialogue, or it only began emerging 
towards the end,” one assistance 
provider said. “There were ... concerns 
about who was telling what to whom. ” 



Documenting the Change Process 

Documentation establishes an historical 
record of the steps, processes, and 
interactions associated with a complex 
initiative. It produces examples and lessons 
that help sites and other interested parties 
understand why they succeeded and where 
they went wrong. It can help researchers 
identify emerging effects and patterns long 
before they are clear enough to measure. 
And it educates foundations and other 
funders about issues and solutions that are 
relevant for strategic planning and program 
monitoring. Based on information gathered 
by the implementation study of the 
Neighborhood and Families initiative, for 
example, the Ford Foundation shifted from 
a centralized system of technical assistance 
for sites to a structure that gave 
communities a stronger voice in designing 
technical assistance services. 

Several of the larger evaluations described 
at the conference included a documentation 
component. The Plain Talk evaluation used 
on-site ethnographers to track what was 
happening in the sites — to understand the 
community context, attend workshops, 
observe meetings, and conduct formal and 
informal interviews. Although the focus of 
this activity was to share findings with a 
national audience, some ethnographers also 
shared their weekly write-ups with sites. 

Similarly, the Neighborhood Partners 
Initiative tried to use documentation to 
encourage sites to think strategically about 
what they were doing. Documenters, who 
also belonged to the initiative’s technical 
assistance team, were expected to 
participate in cross-site activities and 
review locally developed materials. They 
also met frequently with project directors to 
analyze what they had learned as the 
initiative unfolded. 
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Providing Communities with Feedback 

Increasingly, evaluators and other 
researchers are recognizing the value of 
giving the communities they study some 
feedback on what they are learning. The 
message that feedback gives to 
communities, as researcher Robert Chaskin 
summarized, is: Here’s what we’re 
learning about what you have shown us. 
Here are what these local dynamics are 
leading to. Here are the tensions in the 
way priorities are developed. Here are 
tensions in the levels of local participation. 
Here’s what’s playing out when you say you 
want to achieve these goals. 

Sharing research data with communities and 
stakeholders has several benefits. 

Feedback to sites can: 

© Reassure people about the intent of 
research, build local trust, and reduce 
apprehension toward evaluation 

© Establish a local audience for 

information about efforts to improve 
outcomes for children, families, and 
neighborhoods 

© Provide a vehicle for researchers to 
expand their interactions with sites 
and to have candid discussions with 
local partners about their own data 
collection and analysis 

© Offer a reality check for evaluators, 
as sites consider whether the findings 
mesh with their sense of reality 

© Set a short-term agenda for sites to 
follow up on problems identified by 
researchers 

© Help collaborators rethink their 

actions and plan strategically, not just 
explain what they have already 
accomplished; for example, the 



evaluation of the Cleveland Community 
Building Initiative presented sites with 
baseline information on what the 
initiative had accomplished and then 
asked them to consider whether there 
were any alternative or complementary 
paths of community change that the 
initiative could take. 

Evaluation feedback pushed the Community 
Change for Youth Development site in St. 
Petersburg, Florida to consider 
improvements in areas it had not addressed. 
“The report [said], ‘You’re doing a good 
job at activities but there are things central 
to building an infrastructure that you could 
focus on,’” recalled site coordinator Bufus 
Gammons. “That gave us a charge [that] 
we could share with the executive 
committee.” In response, the site leaders 
created a work program for 320 youth and 
began to thing about ways to stimulate 
youth leadership. 

Evaluations of the Neighborhood Partners 
Initiative, Community Change for Youth 
Development, Cleveland Community 
Building Initiative, Jobs-Plus, and 
Neighborhood and Families Initiative all 
included some form of informal or formal 
feedback to sites. Researchers from these 
studies described the following issues and 
methods: 

Feedback should report timely 
findings on a frequent basis. 

Frequent feedback allows sites to take 
proactive steps toward improvement based 
on current findings rather than waiting for 
an annual or final report. Researchers, site 
leaders, and funders all should be aware of 
the snapshot effect, however: Although a 
feedback report captures important 
information, it represents only one slice of 
time. By the time the report comes out, the 
situation may have changed substantially. 
“[Researchers] may say that we really need 
to look at our work program and put more 
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effort in it ... but [in the meantime] that 
effort could have redoubled or changed or 
it could have gotten a lot worse,” 
Gammons noted. 



The most important thing [about the 
feedback reports] was it gave a historical 
perspective. You could read about what 
had happened in the past, and you could 
look at where you are, and you could look 
at the strategic plan.... It really gave you a 
sense of where you were, where you are, 
and also where you ’re going. 

— Bufus Gammons 



Sites should have periodic 
opportunities for people to comment 
on the content, format, and 
usefulness of the data they receive, 
and researchers should respond to the 
comments. 

Several researchers described compromises 
they reached on reporting sensitive topics 
after listening to site leaders’ perspectives. 
For instance, Community Change for 
Youth Development evaluators decided to 
remove a discussion of personnel problems 
from a widely disseminated site memo but 
retained the finding in discussions with site 
leaders and reports to the funder. 

Reports should include some data on 
dimensions common to all sites in the 

initiative. 

Although most feedback should focus on 
site-specific information, site leaders often 
are eager to know how people in other 
communities are interpreting their 
initiative’s directive. Cross-site analysis 
gives sites a broader perspective on shared 
themes. In the evaluation of community 
Change for Youth Development, for 
example, reports that included information 
on the successes or challenges experienced 
at other sites motivated individual 



communities to sustain or improve their 
own efforts. Conducting the cross-site 
analysis also helps researchers step back 
from the sites they work with and regain 
their objectivity. 

Many evaluations do not adequately 
anticipate the need to share data with 
sites. 

When evaluators do not plan ahead to share 
data with site leaders, they fail to establish 
appropriate mechanisms for information 
sharing and may also fail to allocate 
adequate resources. As a result, 
relationships between local stakeholders 
and evaluators can suffer. This was the 
case for evaluations of the Plain Talk and 
Community Change for Youth 
Development initiatives. From the 
beginning, the evaluators assumed that 
since they were conducting a cross-site 
analysis, their job was to provide findings 
only for a national audience, according to 
evaluator Karen Walker. As the evaluation 
and the intervention progressed, however, 
relationships deteriorated between the local 
stakeholders and the evaluators — both the 
on-site researchers (ethnographers) and the 
national consultants who collected data 
from the on-site researchers. 

Participants in the intervention felt that they 
were constantly exposed to the on-site 
researcher’s examination but at the same 
time excluded from the data loop. The 
annual evaluation report, based on visits 
from the national research team, wasn’t 
timely enough to help sites make changes 
based on the research, recalled senior 
consultant Bob Penn. In addition, some on- 
site researchers felt that their views were 
not adequately reflected in the evaluation 
findings or communicated to sites. 

Evaluators of Community Change for 
Youth Development responded to this 
challenge by generating frequent, site- 
specific feedback memos that focused on 
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the issues of interest to each site and 
represented the combined views of local 
and national researchers. Relationships 
with the sites (and among evaluators) 
improved. 



Techniques for Providing Feedback 

Evaluators of Community Change for 

Youth Development and Plain Talk 

suggested the following approaches: 

❖ Target feedback to the specific 
community — not a broader 
audience — so documents can 
include information on sensitive 
issues 

❖ Refer to the documents as memos, 
not reports, to make them less 
threatening. 

❖ Begin each memo by explaining 
how to use and interpret the data. 

❖ Remind readers that the memos 
include speculation and preliminary 
analysis and may draw on 
knowledge from other sites. 

❖ Include cross-site perspectives to 
indicate when an activity is typical 
or unusual, but keep detailed 
comments about other sites to a 
minimum. 

❖ Do not include recommendations. 
Use the documents simply to tell 
sites what researchers are learning. 

❖ Be prepared to accommodate local 
privacy interests. For example, 
Florida’s strict sunshine laws mean 
that any memo might have to be 
shared with reporters, so 
researchers marked every page 
“draft” to protect the site from 
premature public reporting. 



The failure to include resources for 
feedback in an evaluation design can leave 
evaluators without the motivation to collect 
the kinds of data that allow individualized 
recommendations to sites. The Beacons 
evaluation, for example, did not include 
resources for sharing information with 
individual sites. Data collection focused on 
cross-site implementation and outcomes, 
and evaluators did not feel that they had 
collected enough data about individual sites 
to indicate how local partners might use 
evaluation information to improve, 
according to evaluator Warren. 

Understanding and Using Data as an 
Advocacy Tool 

Although data are essential for improving 
program quality, interventions should not 
be driven by data but their aspirations for 
what the community can achieve. “We 
don’t want to do that blindly; we want to be 
informed,” explained Angela Blackwell. 

“If the information causes us to change 
directions we would certainly do so, but we 
are not out here at the will of the data.” 

In an advocacy context, good data help 
collaborators focus goals, gain credibility, 
and become more sophisticated in 
interactions with the systems they hope to 
change. “We want to be able to utilize data 
as one more tool in our advocacy tool box 
to be able to make points, to gather people 
to the table, to be able to mirror back to 
folks that we’re making progress so that 
they will stay on board as we move 
forward,” Blackwell said. 

Data-based advocacy uses data to negotiate 
change and to build partnerships. For 
instance, when the Urban Strategies 
Council in Oakland, California, learned 
that data on school achievement were not 
publicly disaggregated by race, Blackwell 
and her partners negotiated with the school 
district to share the data. This process 
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initiated positive relationships with the 
school superintendent and school research 
department that smoothed the way for later 
advocacy activities. 

Building Local Capacity to Appreciate 
and Use Data 

Because data often are challenging to 
obtain, analyze, and understand, it is 
tempting to simply “buy” data capacity by 
sending experts to sites to do the hard 
work. This approach fails to build the 
capacity of residents and community 
agencies to value information, to gain 
access to it, to understand what conclusions 
it supports, and to use it to compare 
achievements against goals — in short, to 
promote “social learning [about] progress,” 
as Sharon Milligan put it. 

How can the use of data foster new skills in 
communities? Conference participants gave 
the following examples: 

• In the Oakland example provided by 
Blackwell, the exercise of “stumbling 
and fumbling” through data collection 
and analysis was a growth process for 
local collaborators. If the Urban 
Strategies Council had contracted with 
an expert to obtain the disaggregated 
data, “We wouldn’t have [become] 
stronger advocates,” Blackwell said. 

• The reports and feedback memos 
provided by evaluators to Community 
Change for Youth Development 
partners in St. Petersburg, Florida 
offered a template for communicating 
findings that site leaders adopted for 
their own reports. Local leaders thus 
became more effective in reporting 
their goals and progress. 

• Data feedback to Plain Talk leaders in 
New Orleans helped them realize that 



some communities were better 
positioned to deal with certain goals 
than others. Leaders developed new 
priorities, benchmarks, and standards 
for each community to ensure that they 
matched local assets. One community 
focused on schools, another on health 
care, and another on faith-based 
services. “We were able to use [data] 
to change our scope of work,” said site 
director Tammi Fleming. 

® Plain Talk data also helped New 

Orleans partners learn how to formulate 
their own research questions. “The 
more information you give people, the 
better they are able to . . . actually 
participate positively in what’s going 
on,” Fleming said. 

Until recently, building local capacity to 
use data was not a priority for most 
evaluators, in part because the cities and 
foundations that supported multi-site 
initiatives rarely required their sites to 
report many data. Communities had no 
motive for improving data collection and 
analysis, and evaluators had little incentive 
to allocate resources for that purpose. 



[ T]he ability of the local evaluation to draw 
real lessons and to feed back into program 
design ... depends upon the links between 
the program design people at the 
foundation and the evaluat[ors] . Are the 
program design people really interested in 
what the evaluation is finding out, and do 
they see themselves as a conduit [for] 
bringing those lessons back to the local 
site? [Do] they encourage this sort of lesson 
learning on the part of the collaboratives? 

— Virginia Carlson 



Researchers can promote data as a 
community development tool by providing 
feedback to sites, as described above; by 
involving local collaborators in data 
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collection and analysis; and by 
disseminating evaluation data broadly. 

Involving communities in research 

Giving local leaders a chance to comment 
on research data as they emerge, and 
involving local individuals and stakeholder 
groups in data analysis, increases 
communities’ ownership of the research. 
Although several participants suggested that 
most research does not adequately involve 
local participants or build their capacity for 
research and data use, the Jobs-Plus 
evaluation offered one example. This study 
engages local partners as “interpretive 
partners” in analyzing patterns in the 
preliminary research data. As evaluator 
Riccio suggested, part of the challenge is to 
consider questions such as: 

What implications can we draw from 
this information that suggest a way to 
refine the program? Are we learning 
about a need that has gone undetected, 
that we might want to give more 
attention to? Are people doing some 
things we didn ’t realize that we can 
now capitalize on ? 

“Maybe out of that process we’ll get some 
better idea of . . . what kind of data could be 
gotten more routinely,” Riccio said. “You 
know, maybe there’s some system for 
getting administrative records [that] you 
could track yourselves after the 
demonstration was over.” 

Another approach to building local 
participation, used by the Community 
Change for Youth Development evaluation, 
is to establish local evaluation committees 
of key partners — board members, residents, 
business leaders, volunteers, and service 
providers. A committee that involves the 
same people in thinking about evaluation 
for the duration of the research can build 
deep skills among a cross-section of 
influential people, rather than continually 



starting over with new leaders. A third 
approach, used by the Rebuilding 
Communities Initiative, is to gradually 
emphasize local self-evaluation during the 
final years of the evaluation by increasing 
local responsibility for data. 

The barriers to building local capacity for 
data use include (1) a lack of funding for 
technical assistance on this topic and (2) 
lack of time for self- reflection within sites. 
It is much easier to find support for 
concrete activities than for building more 
general skills, participants said. And it is 
never easy for local collaborators, many of 
whom are volunteers, to set aside sufficient 
time to think about research findings and 
learn what to do with them. 

Disseminating evaluation information 

to a broad audience 

Too often, evaluation data end up only in 
the hands of researchers and program 
officers — not the community leaders, 
practitioners, and policy makers on the 
front lines of change. Even if data do 
reach these people, they often are not in 
accessible formats for readers who lack the 
time to pore through dense reports. 

Researchers and funders now realize that 
evaluation data should be disseminated to a 
broad cast of local stakeholders — but who, 
exactly, are those recipients and what 
information should they receive? Just how 
wide should local dissemination of reports 
be? Should the information go only to local 
project leaders or also to the senior staff? 
What about the entire staff, the governing 
board or council, the neighborhood 
residents? How detailed should reports to 
these audiences be? Should each audience 
receive the same report? 

Researchers are still struggling to address 
these questions, and the answers probably 
depend on the goals and relationships 
within each initiative. 
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The Quest for Data Builds Local Capacity 



When the Urban Strategies Council started in Oakland in 1986, we felt that we needed to show 
why poverty was interesting to talk about. We turned immediately to the census data, but when 
we looked at it we realized it didn 7 seem to describe our community at all. We thought people 
would be more interested in infant care, how teenagers were doing, whether poor families were 
making progress, and how kids were doing in school. The census didn 7 say much about that. 

So we asked people about those things, and we ended up with administrative data. The 
administrative data didn 7 tell the whole story either, although it got us closer. But we were 
able to articulate the things that we needed to know. 

As we put that data into a report, we were challenged by how to put a spin on it that would 
create a new conversation. So we looked at the communities where we had concentrated 
poverty, and we looked at what we thought were some of the effects. That took us to a 
discussion about possible solutions. One of the first things we looked at was early childhood 
programs in areas of concentrated poverty, and then at what supports were available for these 
programs. 

Later, the council took on the job of trying to find out if all of the community’s agencies were 
actually serving the same children and families. We took 8,000 students in the public schools 
and compared them with 19 separate programs. We disaggregated the students by race, 
ethnicity, language, grade point, language proficiency, and other characteristics, and we 
grouped the programs by focus — income assistance, prevention, crisis intervention. We could 
ask, for example, whether an Asian student with low language proficiency and frequent 
suspensions was more likely to be known to crisis or income programs. As a result of 
struggling through it ourselves, with support from Metis Associates, we were able to put 
together a real working group that included all of those agency heads looking at this problem. 
We were able to discover things and be the ones to put it out there, building confidence in our 
ability, becoming very important for this process as it went forward. We were ultimately able 
to put out a report that really drove a whole effort that is still continuing. We were able to 
engage in systems change. 

— From the transcript of Angela Blackwell’s featured speech 
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rom the successes and struggles that they shared, conference participants drew three 
broad lessons about implementing and measuring social change: 



o New roles and relationships are needed for local participants, evaluators, and 
technical assistance providers. 

° The dynamics between funders, evaluations, and communities must be improved. 

° Environments and opportunities must be created that support ongoing self-reflection 
in communities. 

Participants also identified new directions for research. This section presents the conclusions 
that participants reached in each of these areas 



Create New Roles and 
Relationships 

All stakeholders should invest time 

and effort in building effective 
partnerships. 

This will entail efforts to help collaborators 
understand the benefits of community 
change and how these benefits relate to 
their own interests. It will also take efforts 
to reach consensus on expectations among 
technical assistance providers and 
foundations before providing assistance to 
sites. When funders’ goals contradict the 
realities faced by local collaborators, 
grantees tend to “fake, hide, and dodge” 
requirements, as one participant put it. 

This exacerbates trust issues. 

Stakeholders will need to foster trust during 
the intervention planning phase. Trust 
comes from opportunities to discuss 
concerns, a shared understanding of 
partners’ interests, recognizable and 
relevant results, and experiences over time 
in working with the same set of partners. 

Initiatives will need to use assistance 
providers who have the flexibility, 



communication skills, and credibility to 
help diverse partners reach consensus — not 
just those with expertise in certain content. 
Evaluators and assistance providers will 
need to improve their own capacity to work 
with communities. They will have to 
become more responsive to local needs and 
circumstances and better able to give 
feedback on progress and outcomes. 

The individuals involved in implementing 
initiatives will need to take more 
responsibility for shaping evaluation and a 
greater stake in the evaluation process. 
Attempts to engage these stakeholders 
should include discussions about the nature 
and purpose of their activities. Community 
members’ roles in evaluation should be 
clearly defined and financially supported. 

Improve Dynamics between 
Funders, Evaluators, and 
Communities 

Funders should clearly define goals 
and expectations for their initiatives 
and promote candid discussions with 
evaluators and grantees that allow 
each to refine their strategies. 
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Communities do not always know the goals 
and expectations of the funders that support 
their interventions. Confusion over these 
expectations makes it hard to define 
outcomes or the process by which they will 
be achieved — two pieces of information that 
are vital to a coherent evaluation design. 
Furthermore, community leaders, their 
evaluators, and funders may not share the 
same understanding of what an evaluation 
should measure and how it should unfold in 
the community. 

The lack of clarity for interventions may be 
caused in part by foundations’ reluctance to 
dictate strategies rather than having 
communities devise their own solutions. 

But the confusion over evaluation goals 
occurs because there is no shared 
understanding of what constitutes an 
evaluation, some conference participants 
suggested. “We’re operating in an 
environment where everything has to be 
evaluated, and the term is interpreted very 
differently by different people,” said 
evaluator Darlene Hasselbring. “[Some 
funders] are expecting agencies to do their 
own outcome analysis. I’ve seen RFPs that 
tell service providers they have to design 
their own random assignment evaluations.” 

In that environment, evaluation can come to 
mean simply compliance with strict 
performance criteria, rather than a 
thoughtful examination of process and 
outcomes. Having some consensus on 
goals and expectations among funders, 
evaluators, and initiatives helps to ensure 
that research provides valuable learning 
opportunities for all stakeholders, 
researchers agreed. 

This is especially important when the 
intervention involves technical assistance, 
an activity that requires local participants to 
be deeply involved in examining their own 
progress. “If the foundation has a learning 
agenda ... that needs to be put on the 
table,” said Prudence Brown, an evaluator 



and member of the technical assistance 
team for the Neighborhood Partners 
Initiative. “There are some sites that just 
may not want to devote this kind of energy 
to that kind of learning.” 

One size does not fit all. 

Just as a single approach to social change 
does not work for all communities, there is 
no generic approach that works for all 
relationships between funders, 
communities, and evaluators. Funders 
need to be flexible enough to tailor their 
requirements to the circumstances of 
individual interventions, so they can 
remove barriers for communities and their 
evaluators, participants said. Extending the 
length of the start-up period, delegating 
more authority to sites, allocating a mixture 
of small and large grants with differing 
expectations, and improving grant 
distribution procedures so that sites receive 
funds more quickly are some of the 
techniques foundations have used to 
respond to local situations. 

Create Environments and 
Opportunities that Support 
Ongoing Reflection 

Funders, evaluators, and assistance 
providers all have a responsibility to 
establish learning communities — local 
forums for reflection and interaction. 

Technical assistance, documentation 
projects, and two-tiered evaluations that 
promote local evaluation all offer important 
learning opportunities, but opportunities are 
not enough. It takes dedicated funding, 
consensus on priorities, and a significant 
time investment by all stakeholders to make 
these opportunities productive. 
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Funders and researchers must 
recognize that the nature of local 
organizations is to be action-oriented 
and reactive, rather than reflective. 

Evaluations and grant-making practices 
need to be sensitive to the cultures of local 
change agents. However, this should not 
stop researchers and funders from 
motivating and encouraging site leaders to 
become more informed, deliberate, and 
strategic than they might be when 
responding to daily events. 

Documentation, feedback, and 
dissemination efforts must be 

improved. 

Evaluators, researchers, and funders still 
have a long way to go to make data 
available and useful. Information has to 
reach a wider audience, including local and 
national change agents, practitioners, policy 
makers, funders, researchers, advocates, 
and media. It has to be relevant to the 
issues that each audience recognizes and 
able to guide readers to issues that are just 
emerging. Information has to be delivered 
in formats that capture attention and 
stimulate action. It has to support both 
innovation and improvement, increase 
understanding, and reduce confusion. And 
it has to integrate many separate pieces of 
knowledge, so that each one builds on the 
last rather than reinventing it. 



New Directions for Research 

Researchers and evaluators should improve 
understanding of the following topics: 

1 . The connection between 

neighborhoods and children’s 
development — in particular, (1) the 



types of neighborhood interventions 
that are most promising for 
strengthening families and improving 
outcomes for children of all ages; and 
(2) ways that low-income children can 
spend their after-school time to develop 
the knowledge, skills, relationships, 
and behavior needed for good outcomes 

2. Ways that residents connect to each 
other and to community institutions 

3 . Concrete methods for budding social 
capital and for helping individuals in 
poor communities gain access to social 
networks 

4. Links between resources and 
outcomes, including the level and type 
of resource needed to achieve specific 
results and the way in which resources 
support outcomes 

5. The effect on families of factors 
outside the neighborhood, including 
social networks and economic forces 

6. The factors and mechanisms that link 
families to neighborhoods, and how 
these mechanisms mediate specific 
outcomes for children and families 

Conference panelist Claudia Coulton 
summarized the views of many participants 
when she explained, “We want to see better 
outcomes for families and children, but it’s 
the improvement of the mediating processes 
that is actually going to get us there.... 

[W]e need to think, as evaluators, about ... 
which of these mediating processes are 
actually being unleashed and are actually 
responsible for what’s occurring in the 
community.” 
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Appendix A: Conference Agenda 



March 11 



10 - 10:45 a.m. 



1 1 - 12:30 p.m. 



12:45 -2 p.m. 



2-2:15 p.m. 
2:15 - 4:45 p.m. 



Charge to the Conference (Tony Cipollone) 

Plan for the Day (Cindy Guy) 

Breakout Sessions 

Neighborhood and Family Initiative 
Cleveland Community Building Initiative 
Rebuilding Communities Initiative 



Lunch 

Keynote Address: Project on Human Development in Chicago 

Neighborhoods (Felton Earls) 



Break 

Panel 1 : Neighborhood/Family Connections 
Robert Halpern, Erickson Institute 
Theodora Ooms, Family Impact Seminar 
Margaret Beale Spencer, University of Pennsylvania 
Margery Turner, Urban Institute 

Discussant: Heather Weiss, Harvard Family Research Project 



March 12 



9:30 - 9:45 a.m. Plan for the Day (Michael Grady) 

10 - 1 1 :30 a.m. Breakout Sessions 

Neighborhood Partners Initiative 

Jobs-Plus 

Beacons Initiative 

Plain Talk and Community Change for Youth Development 

12:30 - 1:15 p.m. Speaker: Angela Blackwell, PolicyLink 

1:30 - 3:30 p.m. Panel 2: Network, Neighbors, and Community Spirit 

Frank Furstenburg, University of Pennsylvania* 

James Johnson, University of North Carolina-Chapel Hill 
William Rohe, University of North Carolina-Chapel Hill 
Discussant: Claudia Coulton, Case Western Reserve University 

3:30 - 3:45 p.m. Closing Remarks (Tony Cipollone) 

’Unable to attend; represented during the discussion by Claudia Coulton 
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Appendix Bs Keynote Address by Angela Blackwell 

Angela Glover Blackwell is Executive Director of PolicyLink, an institution devoted to advancing 
the policies and practices that achieve and sustain child and family well-being within strong 
communities. She is the former Director of the Urban Strategies Council in Oakland, California. 



Thank you very much. I have been really pleased to have spent the last couple of days listening 
and going around to the sessions that I could attend. I’m always frustrated when things are happening 
at the same time, because you sort of want to go to them all. But the choices that I made have been 
good ones. It is not often that I get to sit with people who are concerned about the research and 
evaluation issues. I spend much more time with the practitioners, the funders, and the policy folks. I 
have always enjoyed those opportunities to sit with people who are thinking about some of the research 
and evaluation questions, and I don’t think we do enough of putting the practitioners in the same room 
with the people who are gathering the data. Just like yesterday, when I was listening to the speaker, I 
wanted him to leap forward to the conclusions, I wanted to know what he thought about all of that. 

But, it didn’t stop me from reaching my own conclusions just listening to him. I have always found 
that when I go out and hear people who are doing research that what they’re thinking about has always 
had some applications to what, a decision we had to make tomorrow, and while the researchers have to 
go at their own pace, and have to rely on whatever evidence and proof they need before they’re ready 
to publish findings, for people who have to act tomorrow it’s an opportunity to set up a program. 

There’s an opportunity to influence a policy direction, there’s an opportunity to fund or not fund. It’s 
very useful to hear people who are thinking about the research talk about what they’re puzzling over. 

What I want to talk about today is community building. I am going to get to the topic that’s 
advertised. I’m going to talk about data, but I wanted to talk about some other things first, because in 
listening to the conversation I just thought it might be useful to try to bring from the perspective that 
I’ve had a chance to gather some of the things that are essential about community building, because I’ve 
heard the phrase used a lot. What I’m going to tell you it means isn’t necessarily what it means, but 
I’m going to tell you what I think it means, because there’s at least some consistency about the things 
that I think are important if you buy into that meaning. 

The most important thing about community building is to understand that it is an approach, not 
an agenda. I am often frustrated when I see that people can seem to have it right, they seem to have a 
real understanding of what community building is, and they’re ready to incorporate it and they’re doing 
three things, one of which is community building. Community building I think of as an approach that 
washes across everything that you do. So that whether you’re involved in a substantive agenda having 
to do with teenage pregnancy, or trying to revitalize a community by focusing on housing, whether 
you’re involved in trying to help people move from welfare to work, whether you’re involved in health 
initiative — there’s a community-building approach that you could use to try to achieve the goals that do 
something in addition to achieving the goal that also helps us to be better prepared to achieve the next 
goal tomorrow. 

So, there’s a wonderful reason for everybody to think about trying to understand the community 
building approach no matter what substantive agenda you’re involved in. When I say community 
building, what I’m talking about are things that fit within this definition. There are probably plenty of 
things outside of it that I would also call community building if you called me on it. But, at least within 
this definition of continuous self-renewing efforts by residents and professionals to engage in collective 
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action, and to problem solving, and to enrichment that results in improved lives; new and strengthened 
assets, relationships, and institutions; and new standards and expectations for life in communities. 

Let me go back over that long and convoluted definition. I wish that I could break it down to 
five or 10 words, but I haven’t been able to. But I think each part of what I’ve said is important. 
Continuous self- renewing, that is. certainly important, because what we’re hoping for is that we’re 
building relationships and ways of working that won’t end when this project ends but will go on, and 
deepen, and be a part of the community and be better every time people come back together to work in 
this way and to work on the next thing, residents and professionals. That it’s not residents or 
professionals — and I hope that we will think of both of those terms as being very broad so that all of the 
people in the community, and those people who are connected to community, need to be a part of these 
efforts — it needs to be collective action. We’re not coming together just to say that we were all there, 
but we’re coming together to act together. Acting based on all our interests and all our understanding 
makes our action more authentic, more effective. We want these actions to be aimed, yes, at problem 
solving, but never forgetting about the enrichment piece that we all hear a lot about. We don’t just 
want to focus on problems, we don’t just want deficit approaches, we want to incorporate the assets as 
well. So, problem solving and enrichment, so that when we’re working with young people, when 
we’re working with families, we think about the things that are important for our lives and we 
acknowledge that everybody wants those same things. So, let’s make time for culture, let’s make time 
for fun, let’s make time for relationship building, so we’re enriching lives at the same time that we’re 
focusing on problem solving. 

As a result of bringing all of these people together to act collectively, to problem solve, and to 
enrich, I think the measure of our work always has to be improving lives, that we must never leave that 
out. We do this to improve lives. But at the same time that we’re improving lives we want to 
strengthen relationships, institutions, and those strengths and assets that were there in the community. 
Probably one of the most important things that community building does is it creates new standards and 
expectations for life in the community, whether it has to do with standards for the police department, 
whether it has to do with our expectations from the schools, whether it has to do with our expectations 
of each other and our ability to be able to hold each other accountable. All of that is what I mean when 
I say that we’re doing community building, and clearly that is an approach, not an agenda. You’ve got 
to wrap it around something. You have to be working on something that you can apply this to. 
Therefore, it should be everywhere, not isolated. 

What’s so interesting about community building is it really gets over so many of the divisions 
that have kept us from being able to really work together — those divisions that have to do with whether 
we’re taking a people-based approach or a place-based approach to problem solving. Clearly, 
community building requires that we respond to the challenges and issues affecting people within the 
context of the places where they live, because the only way that we’re really going to have sustainable 
solutions is if the places somehow embody that which is going to help people to move on. We need to 
stop thinking of our work — whether we’re working on health, or education, or employment — as 
something that we do to, for, or with merely individuals, and then they’re gone and the thing that we’ve 
achieved is gone with them, so that we educate people and when they leave the education has gone from 
the community. We get people employed and once they get employed and move on, employment is 
gone from the community. We need to think of these communities as places that have a continuous, 
renewing capacity to educate, to make healthy, to employ, so that those things get embedded in the 
communities so that it’s there for the next wave of people who come through. 
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So, we don’t have to worry over whether our strategies are causing people to leave 
communities. I always think that that’s an odd worry. You know, we want to do the best for people in 
the context of where they live, and if they move on then that’s really what we wanted. We wanted 
them to be able to make choices, and they couldn’t make choices unless we improved the place where 
they live so that they could reasonably choose to stay. But, they can also then reasonably choose to go, 
and we have a place that supports the next person, or the next group of people as they go through. So, 
if we’re doing things in a community-building fashion, there are a couple of things that we have to pay 
attention to. There are a lot of things that we have to pay attention to. I’m only going to talk about a 
couple of them because of time — not because I couldn’t go on and on. 

I want to talk about leadership; the institutional capacity that you need; and capacities that 
communities need to be able to analyze, own and understand, and utilize data. I picked those three 
because I don’t think we pay enough attention to those three, leadership in particular. 

There’s nothing more important for increasing the capacity of a community than to increase the 
capacity of its leadership. Leadership is vital for social change, and I think that very often what 
happens is that we discount leadership. We say that if a program is successful and it had a wonderful 
leader then it was because of that wonderful leader, and we really can’t look at what the impact was of 
all those things that we might have thought of as the treatment, or however we want to label it. Very 
often we assume that we can’t replicate what’s there because of the unique role of leadership. I think 
there is a unique role for leadership and we need to pay attention to it. Let’s think about some of the 
things that we know. I’ll just talk about some of my friends who I think are extraordinary leaders. 
Geoff Canada is one of them in New York. He runs the Rheedlen Center for Children and Families, 
and he’s an amazing person. There are a lot of things that are amazing, but one of the things that is 
essential for the work that he does is he has a broader vision than any vision any foundation could bring 
to his area of Harlem. What’s good about that is that because he has a very big and broad vision that 
predates a foundation or government or research institution agenda, he knows how to bring that in and 
to utilize it for some continuous interest and goal in that community. I think it’s important to find 
leadership that had a broad vision before the money or whatever was dangled. If people only come to it 
because there’s this opportunity, this is not the big vision that you need for the continuous self-renewing 
efforts that we want to have happen in that community. 

The next thing that Geoff Canada has is an extraordinary capacity to inspire, and when you’re 
talking about achieving things that haven’t been achieved before, you need somebody leading that effort 
that inspires people to be able to go on in the face of no evidence that going on is going to get us 
anywhere. But it’s only because you go on that you find out where it can take you, and you need 
leaders who can inspire so that they can bring people in who are doing other things, who have other 
things that they could do to work on this effort, and to get people who don’t see the full picture — and 
don’t necessarily have an experience that allows them to think that this is going to be important — to 
stick with it. 

The next think that Geoff Canada brings is a certain excitement that causes people to invite him 
places. That’s a pretty hard thing to measure, isn’t it? But that is so essential for doing a successful 
project, for doing successful innovation. If people don’t invite you places you don’t get to hear Dr. 
Earls, whom I heard yesterday. You’ve got to get invited so you can meet the people who wouldn’t be 
in your usual circle, so you can make your own connections, not just the connections that your funding 
partner thinks that you ought to make. You have your bigger vision, a broader vision, and you need 
the opportunity to be able to find those people who can help you realize it. You need to have leaders 
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who are running these things who are going to get invited places so they can expand their sense of the 
possibilities. 

Now, I think that probably most of the things I just mentioned are not things that we know how 
to measure or we think about as being essential, but I want to say to you they are absolutely essential 
for trying to do a demonstration. You find leaders who have that broad vision, have the capacity to 
inspire, and have an excitement about them that gets them invited places, why they don’t even know 
they are there. But, when they are there I promise you they will take something away with them that’s 
going to make their project stronger. 

I think we need a different kind of leadership to do it the first time than you need to do it the 
seventh time, and we need to make distinctions between that. To do it the first time, you need 
somebody who’s going to inspire and who’s got a big vision, and who’s going to go out there and do all 
of that reconnaissance to try to put it together with bubble gum and glue and anything else they can 
find. After this has happened several times you need a different kind of leadership, and I have a lot 
more to say about that. 

Let me just move from there, because I have a lot more to say about leadership, to talk about 
institutional capacity to do collaboration. We all understand that, once we embrace the complexity of 
the lives and problems of people whose lives we’re trying to improve, you have to then work 
comprehensively, you have to work across differences, you have to work across disciplines, you have 
to work collaboratively, because no one group can do everything. I don’t really think it’s possible for 
one agency to be able to have a truly comprehensive response to what people need, because you may 
try to work across a lot of areas but you’re not going to be good enough, or deep enough, or connected 
enough to be able to do all of them well. 

So you’re going to have to collaborate. Collaboration that comes out of that kind of insight is 
the collaboration that has a chance at working. Just because you tell people you need to collaborate 
doesn’t mean it will work, but if people really understand what they have to do in order to be able to 
solve the problem in a way that’s going to be sustainable, they will come to the need to work with 
others and to work collaboratively. But it takes a certain kind of institutional capacity to be able to 
support collaboration, and I’m not sure that we have figured it out yet enough to be able to help people 
develop that capacity. I think we need to spend a lot more time thinking about it. 

Part of that capacity is the capacity to really staff the change process in a service mode, because 
it’s that sense of service that allows all of the partners to feel not very threatened by the fact that you’re 
providing this role. It can’t be so much of a serving role that this thing can go wherever it wants. 
You’ve got to have a point of view. You’ve got to have something that you think you’re trying to 
achieve in a certain way. Trying to combine that point of view with that sort of posture of service is 
part of what this institution has to bring. But it also has to bring a lot of capacity, capacity for new 
ideas. If you have an institution playing that role they have to be able to get fed the best ideas that are 
out there, because very often what happens in communities is that ideas from the outside don’t catch on 
because we’re dealing with the “not-invented-here syndrome.” This institution has to have a way to be 
able to bring new ideas but introduce them so they don’t feel foreign. So, being able to allow 
institutions to have lots of different individuals who have their hands and their feet in different areas 
that they could bring back in and translate is very essential. 

The institution also has to have the ability to weave together and cross the lines. The institution 
in local communities has to be able to cross lines of differences. It has to be able to cross lines of 

Angela Blackwell B_4 




56 



differences. You cannot solve the problems in the African-American communities in this country only 
working with people who are African-American and who happen to live in those neighborhoods. You 
have to reach out of the neighborhoods, you have to reach out into different disciplines, you have to 
reach out into different races, you have to reach out into different classes, and the institution has to 
have within it the ability to do that. They have to have the ability to do that, and there’s certain kind of 
individuals that bring that — but it’s not enough just to have individuals. The institution has to have the 
reputation of being able to do that so when people call on the institution they’re prepared for that 
institution to bring people to the table who you might not be comfortable with, but you feel like that 
institution is going to make it safe for you. That’s a tall order for an institution, to be able to be the 
convener who could invite in the people who have been fighting, who’ve been distrustful, who have 
assumptions about each other, and to create a safe space where the hard work can be done. Safe spaces 
don’t require that we don’t deal with tough issues, they don’t require that we agree all the time — we 
could even get angry — but what we have to know is that you are not going to be embarrassed, that you 
are not going to be singled out as the villain when you come to that place. That’s an institutional 
capacity we need to learn more about. 

Another capacity that the institution has to have is to be able to be comfortable with the data. 

We cannot plan effectively if we don’t know what’s going on. We have to figure out ways to get 
people who are working on change in local communities to have a respect for and a comfort with data. 
I’m not sure that buying data capacity for an entity accomplishes that. The data is challenging, it’s 
often hard to get, it’s often not easy to understand, and therefore there’s a temptation to bring in people 
to work with local entities who are comfortable with it, who know how to analyze it, who know where 
to get it, who have relationships. And I think so many things are possible I would never say that that is 
not a model that can work, but I think it is a model that usually doesn’t work. Because what you really 
want is for those entities that are going to be there tomorrow to develop a capacity to be able to access 
this data, to understand this data, to understand what’s not there and what’s not there in relationship to 
where they’re trying to go. 

Let me use a couple of examples from the work that I’m most familiar with to try and make this 
point. It’s the work we did at the Urban Strategies Council. When we first started off trying to do the 
Urban Strategies Council in 1986, we felt that the first thing we needed to do was to tell a story about 
why poverty was interesting to talk about in 1986. This is in Oakland, California, where people hadn’t 
really talked about poverty for a long time. When we thought about putting together some data around 
poverty, we turned immediately to the Census data because there were lay people just trying to think 
about where you find that kind of data. That’s the first thing that came to our minds. When we got the 
Census data and looked at it, we realized it didn’t seem to describe our community at all. The things 
that we thought about in Oakland that we thought people would be interested in [were] infant help, and 
how the teenagers were doing, and whether poor families were making progress, and how kids were 
doing in school. That was the real stuff that people wanted to talk about, and the Census data didn’t 
say much about that. 

Because we wanted more and deeper discussions, we asked people about that, and we ended up 
with administrative data — and that was a good way to get to administrative data, rather than somebody 
coming in with a package of material for us that included the Census data and the administrative data. 
We got to the administrative data because it had the conversation we wanted. When we got the 
administrative data, we found that it didn’t much tell the story either, though it got us closer. But then 
we were able to articulate a whole different set of things that we needed to know that ended putting us 
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in touch with people who could do what I think they call primary research, and begin to find out some 
of the things in some specific areas that let us get even a little deeper. 

As we got all of that data, which we put together in a report called A Chance for Every Child , 
we were challenged by how to be able to put a spin on it that would create a new conversation. That 
was the same time that William Julius Wilson was doing his work on the concentration of poverty. So, 
we grouped it around concentration. We looked at the Census data as he had, looking at 1970, 1980, 
looking at the increase in the concentration of poverty. But then we took the communities where we 
had concentrated poverty, and we looked at what we thought were some of the effects, or at least things 
that were occurring together. That took us — once we finished that report and starting talking with 
communities about the data that had come out — to a discussion around, well, if you’re trying to develop 
solutions, how does this play out? One of the first things we looked at was early childhood programs. 
We looked at early childhood programs in areas of concentrated poverty and tried to look at the kind of 
supports that were available. We did it with the providers of early childhood development supports and 
other community people. 

All the time that we were doing this, the Urban Strategies Council had something it was trying 
to accomplish. I think that it was no accident that right prior to coming to the Urban Strategies Council 
for 10 years I had been a public interest lawyer. I wanted to disclose that because I think it’s important 
to disclose these kinds of things. As we’re looking for people to select for projects, if we’re looking to 
see why they were effective, I think very often we don’t look back at those kinds of issues. I was an 
advocate prior to being a public interest lawyer; I had been a community organizer. Everything I had 
done as an adult had been advocacy work, so when it came to data I never thought that we were going 
to be data driven. I used to tell the staff, “No matter what people on the outside might say — and I don’t 
mind that they say we’re data driven because that gives us a certain amount of credibility — don’t you 
ever think that we are. We are not being driven by data. We are driven by our aspirations of what it is 
we want to achieve for this community.” Now, we don’t want to do that blindly, we want to be 
informed. If the information causes us to change directions we would certainly do so, but we are not 
out here at the will of the data. We are trying to achieve something, and we want to be able to utilize 
data as one more tool in our advocacy tool box to be able to make points, to gather people to the table, 
to be able to mirror back to folks that we’re making progress so that they will stay on board as we 
move forward. 

As we began to get more into the data, we began to be much more sophisticated as advocates. 
Our two biggest advocacy projects both had to do with the schools. When we got involved in trying to 
help all of those people in Oakland who were trying to improve the Oakland public schools, one of the 
first most shocking things that we found out is that the school achievement data had not been 
disaggregated by race. It had not been publicly disaggregated by race, although there had been some 
disaggregation in the research department. I couldn’t believe it. I just assumed, when we went to the 
superintendent’s office in 1989 to ask for all the data disaggregated by race, that we would just be told 
where it was. But it was not that easy. They didn’t do it because they thought that it would cause a 
riot. They thought that if they did disaggregate that data by race and showed the differences between 
black and white, and other racial differences, it would be a reason for the community to be up in arms 
in a way that the school district was not prepared to respond to. 

So we entered into negotiations. We entered into a series of negotiations with the school 
district about how to do it, how to be able to present it in a way that would be productive rather than 
negative. I was really impressed that day about a month to six weeks after the request, when the truck 
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came up to our office with all of this data, disaggregated by race. I was impressed because having been 
a lawyer I knew that it could’ve been a two-year fight trying to get that through discovery. I was 
impressed that we had gotten to the point where we really were working together, we were working 
together on something hard and we were pushing the envelope. We had lived up to our end of the 
bargain, too, in terms of how to be able to put it out responsibly. We came up with lots of different 
ways to present it. We went over that with the superintendent. But let me tell you what happened in 
the meantime. We didn’t have any particular capacity about data, so we had to bring in experts to help 
us understand what we were seeing. So, we got smarter. We hadn’t had a particular relationship with 
the research division, and we had to develop one. That turned out to be a wonderful relationship over 
time as we tried to do other things. It helped us to build a trust relationship with the 
superintendent — and after all, this is about trying to improve the schools, not trying to get the data. 

That relationship with the superintendent was absolutely invaluable. 

Now, if we had just contracted with someone on the outside to try to get us this disaggregated 
data by race we could’ve gotten it, but we couldn’t have done the work we were trying to do. We 
wouldn’t have been stronger advocates to try to achieve something. The stumbling and fumbling that 
we went through was part of an important growth process for us as we went forward. Well, our school 
work really went through many twists and turns, and there are wonderful stories and there are sad 
stories, and there are stories today that are really sad stories in terms of where we are with the school 
district. Let me tell you, we made a lot of progress. The one thing that came out of all that work is 
that the school district became committed to equity. No matter what you hear about Oakland, trust me 
on this. The school district is committed to equity. That Ebonics thing that you all heard about — that 
was about equity. I don’t care what you think about what they tried to do, what you have is a school 
district struggling to look at the children who are the most vulnerable, doing the worst, and trying to 
come up with creative equity strategies. The school district is doing a lot of struggling, but they’re 
committed to equity, and I know part of that commitment came out of disaggregating that data by race 
and making it public. 

The next thing that we did, and I’m going to go through this real quickly because I want to 
answer questions if you have any, is we got involved with the schools in looking at the fact that most of 
the students that the school district was concerned about were known to other programs that were 
concerned about children who were not doing well. We made that assertion, we just made that 
assertion at a meeting that we called where we invited the heads of social services, and probation, and 
health, and the city school district, and the county school district. We called the meeting “The Same 
Client,” and our assertion was that you really are dealing with the same client. At the end of that 
meeting, what was decided is that people weren’t quite sure they wanted us to do a study to be able to 
determine if it was really true. 

The capacity that we had to look at school achievement data did not prepare us for this task. 

The Urban Strategies Council took on the job of trying to find out if all of these agencies were actually 
serving the same children and families. We took 8,000 students in the Oakland public schools and 
compared them with 19 different programs, and we disaggregated the students by race, ethnicity, 
language, grade point, language proficiency, all kinds of things. We grouped the government programs 
by income programs, prevention programs, and crisis programs — putting in things like emergency 
room visits being known to probation and crisis. Things like programs that help teens [were grouped] 
in prevention, and things like AFDC in income. So, they could ask and answer the question, for 
example, whether an Asian student with low language proficiency [who was] often suspended was more 
likely to be known to crisis programs or income programs. It was amazing. We had no capacity to do 
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it. Metis Associates came in — Richard Pargemon — and helped us do it, spent all of his time there 
helping us develop it. It was an amazing relationship, but what I want you to know is as a result of 
struggling through it ourselves with support (and the support always came through us) we were able to 
put together a real working group that included all of those agency heads looking at this problem. We 
were able to discover things and be the ones to put it out there, building confidence in our ability, 
becoming very important for this process as it went forward. We were able ultimately to put out a 
report that really drove a whole effort that is still going on, to be able to engage in systems change. 

I want to end on the data discussion because I think that we need to think about how to measure 
whether or not we’re building capacity in a community to value data, to understand data, to utilize data, 
and to fit data into a much bigger agenda. Thinking, as we look at all of these programs, how to 
articulate the value of leadership and measure whether or not we’re building it and supporting it, and 
how to articulate the value of an institutional capacity to be able to support collaboratives, and whether 
we can measure that. Being able to articulate the role and the value of data as a community 
commodity that is understood, sought, and utilized is also a very important part of the change process. 
Thank you. 
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Appendix C: Keynote Address by Dr. Felton Earls 



Felton Earls is a professor of child psychiatry at Harvard Medical School and a professor of human 
behavior and development at the Harvard School of Public Health. He is Director and Principal 
Investigator of the Project on Human Development in Chicago Neighborhoods. 



It’s very nice to be here. I’m going to spend about 25 minutes telling you about the progress 
the Chicago project is making, and hope that we have about 15 or 20 minutes that is scheduled for 
dialogue — because, trained as a psychiatrist, I’m really a listener. I’m also a musician, so, again I 
listen. What I want to tell you about is the evolution of the project up to this point, and talk about 
theory and analysis and interpretation a bit. I’d like to talk about the limitations of what we’re doing, 
because the limitations are as important as the discoveries are, and then conclude with what I see as the 
next steps in the work that we’re doing. 

The project started as a kind of brainstorming session about 1984, 1985, at the Board of the 
MacArthur Foundation. People like James T. Wilson and Lloyd Olan, and a variety of other 
sociopolitical scientists, were complaining about the fact that we know bits of the elephant in a way, but 
nobody has ever studied the whole elephant. When researchers get together, what they’re doing is 
splicing bits and pieces of knowledge together, and you often run into the fact that — since this wasn’t a 
prefabricated puzzle that was chopped into pieces — when you try to fit these pieces of knowledge 
together they don’t fit. Sampling frames are different, measures are different, the investigators’ 
ideology are different. [The Board] wanted to create a single design that asked questions about human 
development, psychological development, or physical development, as well as context, and do it 
simultaneously, and in the most integrated fashion that one could sort of do this. There was a sense 
that you’d be developing a more holistic picture and one that might convey more accuracy, more insight 
than the pieced-together part. 

To give you some sense of a timeline — because I think that it is important that something like 
this can be held together long enough to pay off — between 1985 and 1990, the discussion simply 
continued. Nothing happened. There were workshops, meetings of various kinds. Some people were 
mainly concerned with money, how much money would it take to do something like that, and if you 
take so much money to do that then what’s going to happen to the money for other people. There were 
very — and this is probably the most difficult thing — there were major problems in getting people to 
work across disciplines. With all of the rhetoric about interdisciplinary research, we found that most 
people were in deep wells; they wanted to get out very often, but when it came to language and 
concepts and so forth, they really weren’t prepared to get out. 

A lot of people were in and out of that process pretty quickly. The group who stayed at the 
table were people who represented about 5 to 10 percent of the people who ever came to the table, and 
for one reason or the other they were the ones who were ready in a way to relieve themselves of their 
discipline and their training — and, you know, in some cases even shred up their M.D. degrees if 
somehow that was not compatible with doing what this more integrated, holistic approach required. 

Between ‘91 and ‘94 we started to work on pilot studies. Once we had a group together there 
was a question of what kind of, what would you measure. The Justice Department wanted us to 
measure violence and the Mental Health Department wanted us to measure psychiatric disorder, and the 
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Educational Department wanted, you know, so it goes like that. What would the input and output be in 
terms of measures? What we came up with during that period of time was two connected studies. One 
would be a community study that would study context, and in order to study context you had to sample 
all people who lived in neighborhoods. You couldn’t just sample parents, because in some 
neighborhoods the majority of th& people are not parents, at least they’re not parents of young children. 
Then we were going to do a second study that was a longitudinal study of children growing up in those 
neighborhoods that we have now characterized from a contextual point of view. Of course, we have to 
sample parents and children of families in that case. These studies would be woven together as tightly 
as possible so that you would do a seamless kind of design. 

In 1995, after thinking about every large city in the United States as a reservoir for this 
study — and sort of getting it down to, interestingly, Baltimore, Los Angeles, and Chicago — we chose 
Chicago, not because the MacArthur Foundation was there but it has demographic features that 
combine a lot of the ways in which American cities have been and will be. So, there are old 
neighborhoods, European, African American neighborhoods that are going through change, and there 
are new immigrant neighborhoods. One of the discoveries that I made in looking at demographics of 
Chicago was to learn that not only had there been a brisk migration from Mexico and Puerto Rico over 
the century — really, over a long period of time — but in the 1990s, when we were looking at the Census 
data, we were seeing (so far as you can trust the Census data) a brisk immigration from Mexico at that 
period of time as well. So we began to sort of get the image of a city that in the millennium — when our 
results were really going to matter, 10 years from 1995— we were beginning to get a picture of a city 
that was black and brown, and that the fertility rate and the migration patterns and the political 
economy were sort of shaping a unique kind of place. Where minority groups, in combination, would 
be a majority and the dispersion of white groups, European groups, would make that group in fact a 
minority. 

Now, that hasn’t anything to do with power structure, but it’s demographics. That changing 
demographics was something that really did characterize American urban life. So, Chicago in that 
sense was broadly representative of what was happening if you amalgamated all cities of the United 
States into one big urban clump. 

Between 1995 and now we have been doing a community study and a longitudinal study 
simultaneously. In the last year or so we have started to produce results from the baseline of this study, 
which was ‘95-‘96, and the baseline consists of a study of all the neighborhoods in Chicago. So we get 
a footprint of what was there and a longitudinal study of 6,000 children who were between the ages of 
birth and age 18. Now, it took a separate sample of about 9,000 people to study 343 neighborhoods. 

One part of the work was to representatively sample across all of these neighborhoods and 
collect data in a uniform way to learn something about neighborhoods. But at the same time, we 
wanted through this separate sample of families and children to begin to learn something about growth 
and development, which required another unique sample of 6,000 people. The methods of this study go 
on from there; we didn’t stop at just studying communities from the point of view of the way residents 
perceived them or behaved in them. We also did extensive videotaping of the physical landscape and 
social interactions as we could capture them by driving through the neighborhoods. We also have done 
an extensive survey of key informants or experts in neighborhoods to learn something about stakeholder 
perceptions to contrast with resident perceptions and so forth. Over the years, this data on communities 
is going to be combined, and from a research point of view we think it’s got about as thick a 
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description of neighborhoods as exists. The challenge is to analyze it and to make sense out of it, and 
of course that’s going to be going on for a long time. 

Let me talk about theory, because in some ways I think that’s the most important thing. When 
we started out, the theory that the sociologists had in our group was the theory of mainly social 
disorganization, and that basically is the theory of poverty if you really take away the word social 
disorganization and think of what it means. What it says is that poor communities — for whatever 
reason — are not able to organize themselves in a way to reach consensual or mutual goals. 
Disorganization reflects the absence of this organization, if you will. What happens is that other 
informal organizations, like gangs and illegal economies, at various times fill in what the correct 
organization or the functional organization would do. We played around with that for a long time and 
learned to distrust social disorganization theory. It’s not that one neighborhood is disorganized and 
another one is organized. It’s differential organization that’s the issue here, and by calling a place 
disorganized you in fact miss the way it is uniquely organized in some ways. 

Then we went on to social capital theory and spent a lot of time thinking about the particular 
aspects of social capital theory, which from Coleman’s point of view include roughly three domains. 
One is the degree to which adults and children in a neighborhood or a school relate to each other. He 
calls that intergenerational closure. Another one is reciprocity, and that is the exchanges between 
people who share this organization; the exchanges could be knowledge as well as resources — I mean 
physical resources, material resources. It could be social support as well. It could even be trust, if you 
think about it in that way. Then the final characteristic is continuity or stability. If you don’t have a 
baseline of continuity, commitment, stability, then presumably the closure and the reciprocity are not 
going to work, because they are going to be undermined by this lack of continuity. When you read 
Coleman you realize that continuity doesn’t mean residential stability, which is kind of interesting: 
There can be a continuity in history and time that doesn’t require that people stay exactly the same 
place. It’s like carrying your language, or your attitudes, or your customs to another place but still 
feeling very connected. Social capital doesn’t have to be geographically bounded, that’s one message 
from that. 

That was also insufficient in our thinking as we were planning this. We started to read 
Bandura, who talks about self-efficacy and collective efficacy. In this case, the idea is that there is an 
action potential. Every organization has to solve problems, and the question is how ready is that 
organization to be mobilized into an active interface with that problem to solve it. Since our interest 
was raising children, we began to think beyond social capital to the notion of collective efficacy to say 
how would people use intergenerational closure, or reciprocity, or continuity to take action. The take- 
action part or the mobilization part is hard to find in Coleman. It’s sort of there, but it’s not drawn out 
as it is in Bandura’s theory of efficacy in a way that really makes the measure of action potential 
explicit. So, the measures that we then designed were measures that took Coleman’s theory as far as it 
went but added this new set of issues about activity. We had to spend a lot of time developing and 
piloting and making ourselves believe that this measure was reliable and valid. Over time, I think we 
did that. 



Now, we get to some results. The first step into the community was to use existing Census 
data to characterize the structural and compositional features of 343 neighborhoods in Chicago. 
Basically that was poverty or wealth, so, we looked at the concentration of affluence as well as the 
concentration of poverty. Second was race, ethnic composition. Chicago, like America, is segregated, 
and so there are African-American neighborhoods and Latino neighborhoods and white neighborhoods. 
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Then there is another way that they’re segregated, which is class. We essentially made a grid of race, 
ethnicity, and class, and were pleased to find that in one place we could fill in most of the cells. So, if 
you look at African-American neighborhoods, we have plenty of poor African-American 
neighborhoods, but we also had 37 middle class African-American neighborhoods and 11 upper class. 
(We call upper class when the mean income gets to be $70,000 to $75,000 and above.) So we could 
uncouple class and race/ethnicity for African Americans, we could approximate it even for Latinos, and 
for whites. But, the one big, fat, prominent zero in our cells was that we could not find poor white 
neighborhoods. Now, we knew that there were poor white families in Chicago, but to the best of our 
demographic analysis they did not look like they were aggregated or segregated in geographical space; 
they were spread out. There was a point at which we started to call back our results, because think 
about the implications of that on child development — a poor family, in a context where there are more 
resources because there are a greater variety of families, obviously has the benefit of neighbors and 
libraries and so on. 

The issue became this: If you know the structural compositional features in neighborhoods, is 
that all you really need to know? Because concentrated poverty will explain violence, it will explain 
substance use, it will explain school dropout, it will explain lots of bad outcomes, adolescent 
pregnancies, and so forth. We took a gamble and said that we didn’t think that structural compositional 
factors were that important. First of all, they’re inaccurate, especially for Latinos. Secondly, they 
don’t tell you anything about mechanisms, they just tell you about who lives there. So if you’re 
interested in other dimensions, then you need to know something about how a neighborhood actually 
functions. That s where the social capital collective efficacy measures became important. It was at that 
point that we measured across 300 neighborhoods and 9,000 people those characteristics from the 
theory that I just mentioned. We were amazed to find out that collective efficacy was a robust predictor 
of violence and that no structural characteristics that we would put into equations could wash out 
collective efficacy. Not only had we shown that it’s important to measure neighborhood mechanisms, 
because that then tells you what to intervene on, but it turns out it was even more a powerful predictor 
than we thought. If you read some of our papers, the equations are really pretty amazing because with 
9,000 people you can do a lot of mathematics. What it means is that you can test the strength of this 
mechanism more powerfully. It’s like a dream come true, in a way. We could take marital status, race 
composition, educational level, household size, point of immigration, where your grandfather and 
father was born, to look at enculturation effects. Despite all of that, this way in which communities are 
organized around kids, the readiness to take action [was important]. 

The questions were, If you saw a child skipping school, do you think people around here would 
do something about it? If a fire station was about to close, do you think people around here would do 
something about it? If you saw a kid disrespecting an older person... and it went on like that — different 
ways in which people in a community perceive each other. This is perception, it’s not behavior, 
attitudes. That perception of the space in which you live becomes this very powerful variable. I think 
that one very interesting insight of this is that you can move from the objective level of analysis — that 
is, what do people actually do, what are their characteristics like race, ethnicity, income, and so 
forth — into this subjective domain of how do people perceive each other. The perceptual measure is as 
strong if not stronger than the objective measures. That, to me, was really a surprise, because you 
usually— I guess being a psychiatrist it wasn’t that much of a surprise. The point is, I’d never seen 
perceptual measures work that way, and so it was gratifying to know that subjective states of experience 
can be precisely measured and then put together with lots of other kinds of data. The way that people 
perceive the world around them becomes a kind of reality in a way, in terms of it controlling events. 
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Now, the next thing that we started to do is that once we had each of these neighborhoods 
outlined, we made a map that had a collective efficacy score for each neighborhood. We got interested 
in spatial analysis, because this is not random, obviously, and started to ask two kinds of questions. 

One is that collective efficacy could have different kinds of political valence associated with it. For 
example, a Latino neighborhood that scored high in collective efficacy might have different properties 
in it. That is, different values are operating to drive up that collective efficacy than if the neighborhood 
was primarily white or primarily African American. A score of 3.5 on this test tells you about level, it 
doesn’t tell you about values. We wanted to find out about values, because you could have a 
neighborhood that was race-exclusive and had very high collective efficacy and low violence within it, 
but much of what it was doing was circling wagons to keep out people who didn’t look like them. You 
could also have industrious communities — and this is what characterizes a lot of Latino neighborhoods, 
where people are really working hard to become naturalized, to become citizens of the United States, to 
do better in this country than they think they could have done in their country of origin. That produces 
high collective efficacy, and in some African-American neighborhoods there was a very high value on 
education, particularly in the middle-class neighborhoods. So, people prided themselves in the fact that 
every child born on this block for the last 20 years has gone to college, for example, and that became a 
kind of value in a way that represented the action potential of that neighborhood around kids. These 
people really sent a message out to kids that education is important. 

Now, here is what we find from the spatial analysis. The danger about neighborhood research 
is that you reify the geographic boundaries of a neighborhood. We started out with Census boundaries, 
understanding that that was an artifact, if you will — it wasn’t real, it was something that the Census 
department did. Also, people give names to geographic units and that is historical, so that if Pilson was 
a Polish neighborhood up until 1970, 1975, but it’s now completely a Mexican/Mexican- American 
neighborhood, it’s still called Pilson. That name has no relevance because the kids in that 
neighborhood go around drawing Mayan and Aztec murals to make it look like Mexico, but it’s still got 
this name of Pilson. What then really does represent neighborhoods as a functional unit when you’re 
trying to analyze how those units impact on human development? 

What we’re finding is that neighborhoods are embedded in larger regimes, and so the notion of 
a school district or a police district actually fits better with political economy definitions than 
neighborhoods do with human development concerns. When you look at the regimes, what you find 
out is that a neighborhood that had low collective efficacy but is in the regime of high collective 
efficacy benefits from that. On the other hand, a neighborhood that had high collective efficacy but 
was in a regime of low collective efficacy is taxed. The spatial analysis, I think, reveals that in 
important ways, and we have some publications that are coming out with interesting maps that show 
this. 



The interesting thing is that this breaks down by race/ethnicity. The neighborhoods that tend to 
be low collective efficacy in a regime of high collective efficacy are white. So, here is a neighborhood 
that’s not functioning very well, but it’s benefiting from other neighborhoods around it — and guess 
what, the neighborhoods of high collective efficacy in regimes of low collective efficacy are black, 
which means that they’re being taxed. So, if you project this into a moving picture, a dynamic picture, 
what you see is vulnerability in the black neighborhoods that are hanging on to staying in the city and 
raising kids in that neighborhood, compared to these white neighborhoods that have a second chance in 
a way, or the kids in those neighborhoods have a second chance because of the surrounding factors. 
That’s about as far as we have gotten. 
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Let me tell you about the limitations, some of which I’ve talked about. One is that we’re still 
not clear about the unit of analysis, so there’s Census tract and [neighborhood] names and so forth, and 
so on. But there are also block groups. And the within-neighborhood variation is very large in all of 
these cases. The greatest statistical challenge we’ve had is how to overcome within-neighborhood 
variation in order to pick up between-neighborhood variation. Do you understand what I mean? You 
look between neighborhoods, and you look at real estate value and you say, Now, there really is 
something different about these two neighborhoods. But when you look microscopically, person by 
person, the within variation is so much from the low real estate to the high real estate neighborhood that 
you almost say, Well, why doesn’t everybody just move in Robert Taylor and save some rent? It’s 
obvious that there are between differences, but overcoming this within problem has really been 
extraordinary. I think we’ve worked it out, but it’s still a problem. At the block group level you can 
find very interesting effects that wash out at the larger level. For example, public housing; some 
neighborhoods have a big public housing complex like Henry Homer, but it’s embedded in a larger 
neighborhood. But that public housing represents a unique ecological niche, and it would be confusing 
to include growing up in the public housing with growing up around the block from public housing. 

The second thing is that we are aggregating — it’s like voting, you know, in a democracy — just 
adding up responses that people give us, and that’s one way of doing social science. There are other 
ways of doing social science that are much harder to do but might be more valid to do, and I’m going 
to get to that. 

A third thing is that this project landed, it parachuted on Chicago, and I’m a little embarrassed 
to say that in front of people like you but the truth is that we made up our minds that we wanted to do 
this big project. We then designed it a little bit and looked for a place to do it, and then we sent the 
helicopters in and landed our research assistants, and they started to collect data. Now, we realize that 
that didn’t work. There was a lot of resistance to getting the study off the ground and launched, and we 
had to back up, pull people back in and do about 18 months of work to make ourselves legitimate. In 
the process of making ourselves legitimate, I think we were all transformed. We came out being a 
different kind of researchers and different kind of people than we were before. But that element of 
participation in the research is one that we still haven’t quite figured out how to do and do well. We 
realize that its something rich and important; in doing it, that’s the way we came out of this process. 

But in terms of how to do it when I write a grant proposal to NIH or someplace like that. . . . 

So, the next steps. The first step is that now that we understand the way communities work — at 
least just roughly — we want to link that to growth trajectories of children in a longitudinal setting. One 
way of looking at this is to take reading scores. We think that in some neighborhoods you’ll have 
reading scores that look like this — they sort of rise up to third grade and then they get flat. In other 
neighborhoods you have reading scores that continue to go up after third grade, and in other 
neighborhoods you have not just flat at the third grade but dipping at the third grade. You can say the 
same thing for propensity for drug use, for propensity for anti-social behavior. There’s a growth 
function on behavior and skills that are, we think, neighborhood-specific. Now that we have this 
community setting, we’re going to link it to these growth curves. 

The second thing is that the field has been turned on its head a little bit in the last 10 years by 
the profound knowledge about how health is affected by socioeconomic status, and it’s just being 
reproduced all over the place. When you can control for everything else — access to health care, et 
cetera — SES effects on mortality, on morbidity, on school achievement, and so forth are profound. We 
live in a tightly parochial civilization where where you stand in the hierarchy is translated rather 
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directly into well-being a number of years later. We have to integrate that into our theory of thinking: 
What is it about social class that gets under a person’s skin? We’re beginning to think developmentally 
about this, to look at prenatal care, how social class affects prenatal care, how it affects cigarette 
smoking and risk behavior, and so forth. We’re going to do that anyway in some ways, but now that 
it’s sort of linked to lifelong risks we’re learning that there’s a developmental programming that may be 
going on early in life that sets you into your social class self-constraints. And you’re not going to move 
very much once you’re on that trajectory. It’s kind of an interesting idea, and it opens a side of life to 
ponder. 

Then the final thing, and I want to stop on this, is deliberation. The contrast to aggregated 
approaches in a democracy are deliberative approaches. One way we make decisions is by having two 
people who have different points of view, a Republican and a Democrat, talk it out. The sum of that is 
not one person voted this way and another person voted that way, it’s the transformation that took place 
in the act of a conversation, in the act of a discussion. That happens to us all the time — we go into a 
situation with one point of view and we come out, after having shared our point of view with somebody 
else’s point of view, in a very different way. Yet in social science we don’t have very good ways of 
capturing deliberative processes, so we always fall back on this aggregation way. And with statistics 
you can do wonders with piles of numbers, but it may not be getting at the most essential ingredient 
that works — that is relevant, I think, to a word that you have in your title: neighborhood 
transformation. 

The final statement, the very final statement I want to make is that in this whole process I think 
we have shifted from what I would call normative academic science to something else. I have to be 
careful about what I call something else, but it’s critical, it’s transformative. The way we look at and 
interpret our data is changing, our expectations of the people who work with us are changing, and I 
think that in the process of that that we’re actually getting to a kind of science that says the purpose is 
the transforming of society, not to aggregate data and somehow explain things, if you will. That’s the 
way I was taught to be a social scientist — you come up with a model, you test a model, then that model 
then explains why the world is like it is. That’s not what we’re doing. I think we really are trying to 
use our framework to engage citizens and stakeholders in a way that transforms the world. Thank you. 
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